-- Logs begin at Thu 2022-10-20 08:19:36 UTC, end at Thu 2022-10-20 09:17:45 UTC. -- Oct 20 08:47:04.757137 dv-cinder-master-3-iscsi systemd[1]: Starting Devstack devstack@keystone.service... Oct 20 08:47:04.764439 dv-cinder-master-3-iscsi devstack@keystone.service[81547]: [uWSGI] getting INI configuration from /etc/keystone/keystone-uwsgi-public.ini Oct 20 08:47:04.766444 dv-cinder-master-3-iscsi devstack@keystone.service[81547]: *** Starting uWSGI 2.0.18-debian (64bit) on [Thu Oct 20 08:47:04 2022] *** Oct 20 08:47:04.766444 dv-cinder-master-3-iscsi devstack@keystone.service[81547]: compiled with version: 10.0.1 20200405 (experimental) [master revision 0be9efad938:fcb98e4978a:705510a708d3642c9c962beb663c476167e4e8a4] on 11 April 2020 11:15:55 Oct 20 08:47:04.766444 dv-cinder-master-3-iscsi devstack@keystone.service[81547]: os: Linux-5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 Oct 20 08:47:04.766444 dv-cinder-master-3-iscsi devstack@keystone.service[81547]: nodename: dv-cinder-master-3-iscsi Oct 20 08:47:04.766444 dv-cinder-master-3-iscsi devstack@keystone.service[81547]: machine: x86_64 Oct 20 08:47:04.766444 dv-cinder-master-3-iscsi devstack@keystone.service[81547]: clock source: unix Oct 20 08:47:04.766444 dv-cinder-master-3-iscsi devstack@keystone.service[81547]: pcre jit disabled Oct 20 08:47:04.766862 dv-cinder-master-3-iscsi devstack@keystone.service[81547]: detected number of CPU cores: 4 Oct 20 08:47:04.766862 dv-cinder-master-3-iscsi devstack@keystone.service[81547]: current working directory: / Oct 20 08:47:04.766862 dv-cinder-master-3-iscsi devstack@keystone.service[81547]: detected binary path: /usr/bin/uwsgi-core Oct 20 08:47:04.769182 dv-cinder-master-3-iscsi devstack@keystone.service[81547]: your processes number limit is 47680 Oct 20 08:47:04.769182 dv-cinder-master-3-iscsi devstack@keystone.service[81547]: your memory page size is 4096 bytes Oct 20 08:47:04.769182 dv-cinder-master-3-iscsi devstack@keystone.service[81547]: detected max file descriptor number: 2048 Oct 20 08:47:04.769182 dv-cinder-master-3-iscsi devstack@keystone.service[81547]: lock engine: pthread robust mutexes Oct 20 08:47:04.769182 dv-cinder-master-3-iscsi devstack@keystone.service[81547]: thunder lock: enabled Oct 20 08:47:04.769182 dv-cinder-master-3-iscsi devstack@keystone.service[81547]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/keystone-wsgi-public.socket fd 4 Oct 20 08:47:04.769182 dv-cinder-master-3-iscsi devstack@keystone.service[81547]: Python version: 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0] Oct 20 08:47:04.807372 dv-cinder-master-3-iscsi devstack@keystone.service[81547]: Python main interpreter initialized at 0x55bd166159c0 Oct 20 08:47:04.807372 dv-cinder-master-3-iscsi devstack@keystone.service[81547]: python threads support enabled Oct 20 08:47:04.807372 dv-cinder-master-3-iscsi devstack@keystone.service[81547]: your server socket listen backlog is limited to 100 connections Oct 20 08:47:04.807372 dv-cinder-master-3-iscsi devstack@keystone.service[81547]: your mercy for graceful operations on workers is 90 seconds Oct 20 08:47:04.807777 dv-cinder-master-3-iscsi devstack@keystone.service[81547]: mapped 403077 bytes (393 KB) for 2 cores Oct 20 08:47:04.807777 dv-cinder-master-3-iscsi devstack@keystone.service[81547]: *** Operational MODE: preforking *** Oct 20 08:47:04.808062 dv-cinder-master-3-iscsi devstack@keystone.service[81547]: *** uWSGI is running in multiple interpreter mode *** Oct 20 08:47:04.808062 dv-cinder-master-3-iscsi devstack@keystone.service[81547]: spawned uWSGI master process (pid: 81547) Oct 20 08:47:04.808262 dv-cinder-master-3-iscsi systemd[1]: Started Devstack devstack@keystone.service. Oct 20 08:47:04.809907 dv-cinder-master-3-iscsi devstack@keystone.service[81547]: spawned uWSGI worker 1 (pid: 81561, cores: 1) Oct 20 08:47:04.809907 dv-cinder-master-3-iscsi devstack@keystone.service[81547]: spawned uWSGI worker 2 (pid: 81562, cores: 1) Oct 20 08:47:04.809907 dv-cinder-master-3-iscsi devstack@keystone.service[81547]: running "unix_signal:15 gracefully_kill_them_all" (master-start)... Oct 20 08:47:09.057808 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 20 08:47:09.057808 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: from cryptography.utils import int_from_bytes Oct 20 08:47:09.058619 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 20 08:47:09.058619 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: from cryptography.utils import int_from_bytes Oct 20 08:47:09.059035 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 20 08:47:09.059035 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: from cryptography.utils import int_from_bytes Oct 20 08:47:09.059139 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 20 08:47:09.059139 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: from cryptography.utils import int_from_bytes Oct 20 08:47:09.108436 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Oct 20 08:47:09.108628 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Oct 20 08:47:09.108724 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Oct 20 08:47:09.108813 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] command line args: [] {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Oct 20 08:47:09.108891 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Oct 20 08:47:09.109019 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] command line args: [] {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Oct 20 08:47:09.109102 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Oct 20 08:47:09.109232 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] ================================================================================ {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Oct 20 08:47:09.109382 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Oct 20 08:47:09.109456 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] ================================================================================ {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Oct 20 08:47:09.109525 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] admin_token = **** {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.109699 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] admin_token = **** {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.109789 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] config_dir = [] {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.109867 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.109939 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] config_dir = [] {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.110050 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] config_source = [] {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.110181 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.110265 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] debug = True {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.110354 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] config_source = [] {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.110436 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.110652 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] debug = True {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.110731 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.110826 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.110893 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.110952 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.111080 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.111157 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.111321 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.111403 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.111571 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.111647 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] list_limit = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.111775 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] log_config_append = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.111841 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] list_limit = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.111951 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.112027 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] log_config_append = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.112125 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] log_dir = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.112199 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.112308 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] log_dir = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.112396 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] log_file = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.112464 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] log_file = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.112531 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.112695 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.112793 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.112967 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.113069 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.113196 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.113299 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.113364 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.113429 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.113560 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.113646 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.113770 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.113848 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.114050 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.114134 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.114294 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.114394 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.114553 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.114630 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.114790 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.114875 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.115021 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.115105 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.115255 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.115333 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.115495 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.115572 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.115742 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.115823 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.116019 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.116233 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.36/identity {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.116412 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] publish_errors = False {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.116493 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.36/identity {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.116640 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.116718 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] publish_errors = False {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.116871 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.116995 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.117080 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.117151 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.117280 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.117360 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.117496 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.117586 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.117726 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] standard_threads = False {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.117810 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.117952 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.118036 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] standard_threads = False {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.118185 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.118262 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.118357 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.118439 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.118543 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] use_journal = False {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.118622 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.118744 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] use_json = False {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.118823 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] use_journal = False {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.118902 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] use_stderr = False {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.118973 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] use_json = False {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.119043 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] use_syslog = False {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.119112 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] use_stderr = False {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.119252 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.119332 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] use_syslog = False {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.119497 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 20 08:47:09.119574 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.119730 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.119807 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.119982 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.120066 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.120214 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.120291 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.120461 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.120542 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.120698 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.120778 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.120942 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.121019 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.121174 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.121257 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.121414 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.121497 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.121655 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.121731 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.121899 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.121977 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.122129 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.122208 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.122369 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.122445 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.122622 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.122699 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.122865 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.122951 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.123109 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.123191 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.123344 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.123423 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.123587 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.123665 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.123833 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.123911 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.124072 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.124150 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.124299 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.124373 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.124537 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.124620 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.124791 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.124876 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.125038 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.125116 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.125283 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.125367 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.125511 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.125600 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.125738 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.125816 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.125964 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.126049 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.126196 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.126281 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.126435 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.126533 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.126706 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.126791 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.126937 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.127023 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.127164 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.127248 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.127397 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.127478 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.127639 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.127727 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.127872 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.127956 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.128113 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.128191 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.128338 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.128423 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.128570 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.128703 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.128797 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.128870 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.129016 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.129103 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.129242 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.129358 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.129468 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.129546 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.129743 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.129901 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.129996 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.130070 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.130233 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.130313 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.130491 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.130584 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.130732 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.130809 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.130958 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.131041 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.131191 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.131277 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.131421 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.131499 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.131657 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.131736 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.131882 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.131969 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.132117 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.132229 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.132350 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.132455 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.132583 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.132696 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.132816 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.132921 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.133051 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.133140 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.133264 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.133394 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.133544 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.133632 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.133718 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.133791 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.133944 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.134064 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.134171 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.134264 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.134416 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.134527 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.134681 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.134761 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.134903 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.134984 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.135131 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.135213 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.135362 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.135444 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.135603 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.135683 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.135848 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.135926 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.136093 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.136172 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.136334 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.136417 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.136566 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.136685 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.136799 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.136880 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.137033 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.137113 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.137270 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.137358 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.137506 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.137677 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.137761 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.137841 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.137970 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.138049 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.138204 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.138284 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.138443 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.138575 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.138701 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.138779 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.138943 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.139089 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.139213 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.139301 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.139375 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.139445 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.139602 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.139682 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.139841 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.139920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.140084 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.140161 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.140329 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.140407 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.140567 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.140654 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.140802 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.140881 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.141040 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.141118 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.141266 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.141341 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.141499 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.141577 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.141727 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.141836 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.141957 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.142050 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.142183 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.142262 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.142411 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.142507 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.142666 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.142758 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.142894 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.142974 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.143125 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.143208 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.143354 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.143441 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.143591 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.143669 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.143823 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.143902 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.144069 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.144154 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.144325 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.144403 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.144562 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.144648 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.144790 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] auth.external = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.144879 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.145017 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.145106 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] auth.external = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.145791 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.145888 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.146056 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.146135 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.146283 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] auth.password = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.146372 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.146527 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] auth.token = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.146611 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] auth.password = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.146766 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.146845 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] auth.token = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.147003 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.147094 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.147237 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.147317 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.147461 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.147539 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.147705 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.147792 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.147949 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.148036 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.148165 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.148302 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.148379 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] credential.caching = True {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.148459 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.148627 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.148707 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] credential.caching = True {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.148872 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.148959 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.149117 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.149206 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.149360 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.149439 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.149602 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.149682 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.149834 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.149921 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.150072 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.150151 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.150318 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.150399 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.150574 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.150670 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.150825 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.150911 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.151078 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.151157 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.151334 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.151426 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.151593 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.151673 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.151832 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.151918 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.152085 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.152177 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.152316 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] federation.caching = True {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.152395 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.152550 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.152629 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] federation.caching = True {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.152791 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.152877 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.153033 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.153119 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.153262 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.153342 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.153501 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.153587 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.153732 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.153810 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.153976 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.154055 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.154210 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.154290 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.154460 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.154552 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.154716 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.154795 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.154963 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.155080 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.155196 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] identity.caching = True {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.155276 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.155434 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] identity.caching = True {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.155513 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.155677 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.155758 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.155908 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.155988 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.156139 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.156218 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.156374 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.156454 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.156598 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.156677 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.156839 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.156918 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.157078 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.157157 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.157323 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.157402 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.157551 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.157632 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.157780 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.157860 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.158006 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.158086 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.158236 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.158314 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.158529 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.158614 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.158746 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.158825 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.158998 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.159076 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.159240 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.159321 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.159496 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.159575 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.159733 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.159811 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.159971 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.160054 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.160198 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.160323 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.160434 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.160514 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.160663 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.160744 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.160942 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.161559 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.161691 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.161771 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.161937 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.162019 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.162180 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.162296 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.162565 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.162647 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.162806 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.162887 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.163056 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.163178 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.163291 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.163371 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.163534 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.163614 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.163777 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.163892 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.164007 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.164096 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.164246 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.164332 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.164474 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.164554 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.164708 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.164795 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.164942 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.165021 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.165180 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.165259 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.165415 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.165502 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.165651 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.165731 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.165889 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.165968 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.166132 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.166219 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.166361 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.166441 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.166628 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.166709 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.166865 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.166958 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.167118 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.167198 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.167349 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.167428 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.167582 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.167669 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.167816 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.167904 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.168044 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] ldap.user = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.168124 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.168275 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] ldap.user = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.168362 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.168516 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.168596 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.168746 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.168824 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.168990 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.169078 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.169233 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.169314 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.169475 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.169555 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.169705 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.169793 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.169932 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.170012 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.170162 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.170240 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.170392 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.170547 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.170673 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.170803 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.170931 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.171059 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.171187 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.171326 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.171456 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.171584 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.171712 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.171840 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.171981 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.172120 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.172249 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.172377 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.172506 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.172638 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.172761 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.172898 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.173027 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.173156 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.173284 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.173412 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.173541 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.173676 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.173815 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.173944 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.174071 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.174200 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.174329 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.174465 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.174571 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.174701 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.174785 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.174858 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.174930 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.175014 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.175089 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.175160 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.175231 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.175302 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.175374 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.175453 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.175526 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.175597 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.175669 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.175739 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.175842 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.175931 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.176073 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.176153 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.176308 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.176387 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.176566 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] resource.caching = True {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.176660 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.176812 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.176893 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] resource.caching = True {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.177061 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.177140 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.177294 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.177388 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.177539 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.177619 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.177785 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.177864 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.178017 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.178108 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.178258 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.178337 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.178507 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.178591 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.178742 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.178819 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.178969 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] role.caching = True {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.179049 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.179211 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] role.driver = sql {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.179290 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] role.caching = True {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.179440 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.179521 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] role.driver = sql {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.179677 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.179755 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.179920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.179999 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.180166 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.180247 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.180414 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.180526 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.180660 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.180753 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.180906 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.180987 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.181149 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.181227 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.181394 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.181473 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.181624 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.181705 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.181859 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.181938 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.182102 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.182187 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.182345 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.182427 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.182611 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.182694 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.182861 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.183001 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.183236 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.183318 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.183479 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.183557 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.183724 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.183803 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.183972 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.184053 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.184217 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.184297 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.184449 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.184537 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.184693 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.184772 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.184939 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.185018 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.185177 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.185267 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.185405 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.185488 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.185634 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.185713 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.185865 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.185951 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.186099 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.186179 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.186351 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.186428 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.186620 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.186715 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.186858 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.186937 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.187091 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.187170 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.187318 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.187408 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.187549 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.187628 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] token.caching = True {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.187781 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.187860 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] token.caching = True {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.188018 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.188104 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.188247 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.188326 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.188493 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.188571 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.188734 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.188820 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.188968 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.189048 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.189210 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.189290 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.189451 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.189539 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.189694 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.189773 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.189929 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.190008 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.190165 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.190262 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.190394 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.190520 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.190660 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.190738 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.190911 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.191004 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.191157 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.191245 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.191396 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.191478 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.191637 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.191725 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.191899 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] database.connection = **** {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.191978 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.192151 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.192238 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] database.connection = **** {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.192428 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.192508 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.192664 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.192744 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.192900 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.192982 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.193131 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.193211 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.193364 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.193443 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.193597 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.193678 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.193829 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.193909 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.194066 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.194147 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.194298 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.194379 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.194559 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.194644 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.194791 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.194871 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.195026 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.195107 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.195255 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.195334 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.195497 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.195576 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.195725 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.195843 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.195961 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.196041 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.196191 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.196271 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.196352 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81561) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Oct 20 08:47:09.196426 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 20 08:47:09.196584 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81562) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Oct 20 08:47:09.226439 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81561) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 20 08:47:09.226690 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81561) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 20 08:47:09.226912 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81561) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 20 08:47:09.227037 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81562) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 20 08:47:09.227269 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81562) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 20 08:47:09.227495 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81562) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 20 08:47:09.235145 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81561) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 20 08:47:09.235823 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81562) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 20 08:47:09.262855 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81561) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 20 08:47:09.263079 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81561) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 20 08:47:09.263729 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81562) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 20 08:47:09.263965 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81562) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 20 08:47:09.315857 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81561) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 20 08:47:09.316600 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81562) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 20 08:47:09.364393 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81561) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 20 08:47:09.364622 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81561) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 20 08:47:09.365475 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81562) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 20 08:47:09.365712 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81562) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 20 08:47:09.519494 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81562) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 20 08:47:09.519687 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81561) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 20 08:47:09.519777 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81562) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 20 08:47:09.519857 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81561) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 20 08:47:09.519958 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81562) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 20 08:47:09.520039 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81561) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 20 08:47:09.520146 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81562) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 20 08:47:09.520275 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81561) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 20 08:47:09.520349 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81562) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 20 08:47:09.520446 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81561) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 20 08:47:09.521101 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81562) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 20 08:47:09.521169 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81561) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 20 08:47:09.543857 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81562) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 20 08:47:09.544005 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81561) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 20 08:47:09.544094 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81562) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 20 08:47:09.544232 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81561) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 20 08:47:09.544317 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81562) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 20 08:47:09.544451 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81561) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 20 08:47:09.544535 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81562) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 20 08:47:09.544664 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81561) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 20 08:47:09.544758 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81562) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 20 08:47:09.544873 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81561) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 20 08:47:09.544957 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81562) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 20 08:47:09.545079 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81561) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 20 08:47:09.545163 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81562) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 20 08:47:09.545285 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81561) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 20 08:47:09.556420 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81562) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 20 08:47:09.556536 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81561) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 20 08:47:09.569581 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81561) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 20 08:47:09.569760 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81561) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 20 08:47:09.569872 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81562) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 20 08:47:09.569974 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81561) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 20 08:47:09.570110 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81562) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 20 08:47:09.570194 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81561) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 20 08:47:09.570336 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81562) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 20 08:47:09.570420 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81561) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 20 08:47:09.570587 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81562) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 20 08:47:09.570674 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81561) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 20 08:47:09.570805 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81561) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 20 08:47:09.570888 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81562) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 20 08:47:09.571016 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81562) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 20 08:47:09.571232 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81562) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 20 08:47:09.588171 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.588376 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 20 08:47:09.588523 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.588714 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.588798 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 20 08:47:09.588914 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 20 08:47:09.589064 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.589184 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.589268 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 20 08:47:09.589445 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.589535 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.589700 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.589782 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.589952 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.590033 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.590266 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.599554 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.599830 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.599913 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.600099 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.600182 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.600380 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.600466 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.600689 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.607674 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81561) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 20 08:47:09.607760 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81562) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 20 08:47:09.610564 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81561) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 20 08:47:09.610651 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81562) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 20 08:47:09.610847 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.610931 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.611125 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.611209 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.611409 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.611494 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.611690 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.611772 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.611947 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.612039 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.612210 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.612294 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.612476 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.612557 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.612740 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.612823 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.613010 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.613092 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.613274 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.613356 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.636187 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.636384 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.637629 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81561) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 20 08:47:09.637851 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81562) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 20 08:47:09.637940 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.638135 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.642618 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81561) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 20 08:47:09.642832 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81562) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 20 08:47:09.642916 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.643118 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.643214 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.643403 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.649993 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81561) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 20 08:47:09.650276 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81562) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 20 08:47:09.650361 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.650587 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.655855 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81561) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 20 08:47:09.656149 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.656233 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81562) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 20 08:47:09.656434 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.656517 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.656717 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.656801 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.656992 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.657076 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.657270 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.657361 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.657540 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.657632 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.657805 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.657887 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.658094 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.673076 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.673352 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.673526 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.673798 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.677288 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81561) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 20 08:47:09.677807 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81562) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 20 08:47:09.680463 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.680758 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.681010 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.681311 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.684882 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81561) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 20 08:47:09.685466 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81562) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 20 08:47:09.687945 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81561) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 20 08:47:09.688552 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81562) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 20 08:47:09.691126 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.691425 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.691709 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.691792 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.691996 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.692079 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.692788 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.692874 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.693066 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.693153 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.693381 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.693663 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.709327 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81561) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 20 08:47:09.709610 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.709873 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.710063 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81562) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 20 08:47:09.710146 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.710355 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.710651 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.710920 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.716929 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.717749 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.718461 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.719322 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.719990 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.720263 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.720855 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.721133 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.723002 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81561) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 20 08:47:09.723280 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.723585 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.723838 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81562) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 20 08:47:09.723921 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.724123 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.724237 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.724409 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.724705 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.725095 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.735235 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81561) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 20 08:47:09.735525 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.735804 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.736088 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.736244 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81562) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 20 08:47:09.736461 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.736546 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.736744 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.736834 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.737020 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.737110 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.737439 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.737731 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.738021 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.752252 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81561) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 20 08:47:09.753436 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81562) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 20 08:47:09.755716 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81561) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 20 08:47:09.756906 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81562) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 20 08:47:09.758603 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.759820 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.760109 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.761276 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.761548 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81561) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 20 08:47:09.761827 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.762231 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.762771 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81562) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 20 08:47:09.763058 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.763461 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.768591 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.769858 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.770036 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81561) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 20 08:47:09.771890 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81562) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 20 08:47:09.773700 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.774089 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.774373 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.774677 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.774798 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.775199 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.775487 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.775769 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.782371 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81561) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 20 08:47:09.782687 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.782972 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.783569 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81562) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 20 08:47:09.783865 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.784151 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.789828 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81561) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 20 08:47:09.790105 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.790378 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.790680 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.790949 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.791082 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81562) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 20 08:47:09.791232 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.791368 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.791500 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.791653 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.791768 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.791931 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.792041 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.792204 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.792294 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.792476 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.792566 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.792750 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.792831 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.793021 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.793102 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.793303 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.793392 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81561) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.793590 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.793863 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.794127 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.794391 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.794807 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81562) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 20 08:47:09.828127 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x55bd166159c0 pid: 81561 (default app) Oct 20 08:47:09.829976 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x55bd166159c0 pid: 81562 (default app) Oct 20 08:47:09.831557 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9478db7-92ce-4747-acf5-34dc3d1ae564 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:09.831838 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9478db7-92ce-4747-acf5-34dc3d1ae564 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:09.832103 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9478db7-92ce-4747-acf5-34dc3d1ae564 None None] PATH_INFO: `/v3` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:09.836280 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1/1] 10.222.0.36 () {50 vars in 791 bytes} [Thu Oct 20 08:47:09 2022] GET /identity/v3/ => generated 255 bytes in 8 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:47:16.522322 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c0abf2-6638-4ac0-bc73-e6500468a258 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:16.522870 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c0abf2-6638-4ac0-bc73-e6500468a258 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:16.522870 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c0abf2-6638-4ac0-bc73-e6500468a258 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:16.527141 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1/2] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:47:16 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 20 08:47:16.532314 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a0b012-df63-4c4d-bd41-128b4f429c3f None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:16.532736 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a0b012-df63-4c4d-bd41-128b4f429c3f None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:16.532874 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a0b012-df63-4c4d-bd41-128b4f429c3f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:16.602049 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: INFO dbcounter [None req-f1a0b012-df63-4c4d-bd41-128b4f429c3f None None] Registered counter for database keystone Oct 20 08:47:16.621310 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG oslo_db.sqlalchemy.engines [None req-f1a0b012-df63-4c4d-bd41-128b4f429c3f None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81561) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Oct 20 08:47:16.627640 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writer thread running {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Oct 20 08:47:16.805188 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG passlib.handlers.bcrypt [None req-f1a0b012-df63-4c4d-bd41-128b4f429c3f None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81561) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Oct 20 08:47:16.805490 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG passlib.handlers.bcrypt [None req-f1a0b012-df63-4c4d-bd41-128b4f429c3f None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81561) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Oct 20 08:47:16.863161 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-f1a0b012-df63-4c4d-bd41-128b4f429c3f None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:47:16.865325 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-f1a0b012-df63-4c4d-bd41-128b4f429c3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:47:16.992022 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 2/3] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:47:16 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 461 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 20 08:47:17.405515 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c0abf2-6638-4ac0-bc73-e6500468a258 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:17.474820 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: INFO dbcounter [None req-d4c0abf2-6638-4ac0-bc73-e6500468a258 None None] Registered counter for database keystone Oct 20 08:47:17.495608 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG oslo_db.sqlalchemy.engines [None req-d4c0abf2-6638-4ac0-bc73-e6500468a258 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81562) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Oct 20 08:47:17.501659 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writer thread running {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Oct 20 08:47:17.617713 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c0abf2-6638-4ac0-bc73-e6500468a258 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:17.621927 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc6ff360-a18c-468d-b58c-6ed273d12790 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:17.622564 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc6ff360-a18c-468d-b58c-6ed273d12790 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:17.622841 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc6ff360-a18c-468d-b58c-6ed273d12790 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:17.623108 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc6ff360-a18c-468d-b58c-6ed273d12790 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:17.631223 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc6ff360-a18c-468d-b58c-6ed273d12790 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:17.683749 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: WARNING keystone.common.rbac_enforcer.enforcer [None req-bc6ff360-a18c-468d-b58c-6ed273d12790 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Oct 20 08:47:17.684087 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc6ff360-a18c-468d-b58c-6ed273d12790 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:17.693923 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 2/4] 10.222.0.36 () {58 vars in 1324 bytes} [Thu Oct 20 08:47:17 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 289 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 20 08:47:17.699861 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1a0b012-df63-4c4d-bd41-128b4f429c3f None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:17.717283 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1a0b012-df63-4c4d-bd41-128b4f429c3f None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:17.721700 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87385df6-f5cf-46c4-b9c8-6b0f4a439051 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:17.722826 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87385df6-f5cf-46c4-b9c8-6b0f4a439051 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:17.723109 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87385df6-f5cf-46c4-b9c8-6b0f4a439051 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:17.723378 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87385df6-f5cf-46c4-b9c8-6b0f4a439051 admin admin] PATH_INFO: `/v3/projects/602743e1ebcd495bbe3a73b9f8abba4e` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:17.724973 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87385df6-f5cf-46c4-b9c8-6b0f4a439051 admin admin] RBAC: Authorizing `identity:get_project(project_id=602743e1ebcd495bbe3a73b9f8abba4e)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:17.778598 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: WARNING keystone.common.rbac_enforcer.enforcer [None req-87385df6-f5cf-46c4-b9c8-6b0f4a439051 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Oct 20 08:47:17.778950 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87385df6-f5cf-46c4-b9c8-6b0f4a439051 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:17.783419 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 3/5] 10.222.0.36 () {56 vars in 1207 bytes} [Thu Oct 20 08:47:17 2022] GET /identity/v3/projects/602743e1ebcd495bbe3a73b9f8abba4e => generated 344 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:47:19.883273 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd80d96-2996-4840-80fc-7e03c8449ed6 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:19.883766 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd80d96-2996-4840-80fc-7e03c8449ed6 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:19.883847 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd80d96-2996-4840-80fc-7e03c8449ed6 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:19.887908 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 3/6] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:47:19 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 20 08:47:19.893364 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add70d1c-d038-48db-8026-562177db9d8e None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:19.893770 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add70d1c-d038-48db-8026-562177db9d8e None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:19.893911 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add70d1c-d038-48db-8026-562177db9d8e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:19.913605 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-add70d1c-d038-48db-8026-562177db9d8e None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:47:19.915122 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-add70d1c-d038-48db-8026-562177db9d8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:47:19.922702 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 4/7] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:47:19 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 20 08:47:20.196750 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fd80d96-2996-4840-80fc-7e03c8449ed6 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:20.208820 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fd80d96-2996-4840-80fc-7e03c8449ed6 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:20.212931 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b321ec5-ff48-4311-86d4-3cb728990cc3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:20.213512 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b321ec5-ff48-4311-86d4-3cb728990cc3 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:20.213787 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b321ec5-ff48-4311-86d4-3cb728990cc3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:20.214053 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b321ec5-ff48-4311-86d4-3cb728990cc3 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:20.221431 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b321ec5-ff48-4311-86d4-3cb728990cc3 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:20.223209 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b321ec5-ff48-4311-86d4-3cb728990cc3 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:20.232198 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 4/8] 10.222.0.36 () {58 vars in 1324 bytes} [Thu Oct 20 08:47:20 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 20 08:47:20.237321 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add70d1c-d038-48db-8026-562177db9d8e None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:20.247030 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add70d1c-d038-48db-8026-562177db9d8e None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:20.251332 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee6c86e5-0b42-4525-b367-abba50a03335 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:20.251948 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee6c86e5-0b42-4525-b367-abba50a03335 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:20.252227 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee6c86e5-0b42-4525-b367-abba50a03335 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:20.252500 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee6c86e5-0b42-4525-b367-abba50a03335 admin admin] PATH_INFO: `/v3/users/9804008bc9774f7b96bc93c72ddebd2f` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:20.253792 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee6c86e5-0b42-4525-b367-abba50a03335 admin admin] RBAC: Authorizing `identity:get_user(user_id=9804008bc9774f7b96bc93c72ddebd2f)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:20.255703 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee6c86e5-0b42-4525-b367-abba50a03335 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:20.257200 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 5/9] 10.222.0.36 () {56 vars in 1198 bytes} [Thu Oct 20 08:47:20 2022] GET /identity/v3/users/9804008bc9774f7b96bc93c72ddebd2f => generated 247 bytes in 20 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:47:22.422721 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f500a0f4-ac9c-49f2-a189-83f41ee894da None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:22.423280 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f500a0f4-ac9c-49f2-a189-83f41ee894da None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:22.423280 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f500a0f4-ac9c-49f2-a189-83f41ee894da None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:22.424599 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 5/10] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:47:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:47:22.429708 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83633c6a-e7ec-4f57-ab23-480d5c72ed87 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:22.430153 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83633c6a-e7ec-4f57-ab23-480d5c72ed87 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:22.430360 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83633c6a-e7ec-4f57-ab23-480d5c72ed87 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:22.448932 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-83633c6a-e7ec-4f57-ab23-480d5c72ed87 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:47:22.450117 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-83633c6a-e7ec-4f57-ab23-480d5c72ed87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:47:22.485676 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62eae4c8-403a-4312-ace7-08431f4aaf82 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:22.485961 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62eae4c8-403a-4312-ace7-08431f4aaf82 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:22.486235 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62eae4c8-403a-4312-ace7-08431f4aaf82 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:22.487577 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 6/11] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:47:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 20 08:47:22.492716 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04280d46-822d-402b-907c-d121c68e8b3f None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:22.492997 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04280d46-822d-402b-907c-d121c68e8b3f None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:22.493270 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04280d46-822d-402b-907c-d121c68e8b3f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:22.533595 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 6/12] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:47:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 20 08:47:22.535385 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG passlib.handlers.bcrypt [None req-04280d46-822d-402b-907c-d121c68e8b3f None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81562) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Oct 20 08:47:22.535794 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG passlib.handlers.bcrypt [None req-04280d46-822d-402b-907c-d121c68e8b3f None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81562) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Oct 20 08:47:22.585246 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-04280d46-822d-402b-907c-d121c68e8b3f None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:47:22.586853 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-04280d46-822d-402b-907c-d121c68e8b3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:47:22.684174 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 7/13] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:47:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 20 08:47:22.801448 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83633c6a-e7ec-4f57-ab23-480d5c72ed87 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:22.827207 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:22.905224 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83633c6a-e7ec-4f57-ab23-480d5c72ed87 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:22.937763 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b368a796-a022-47df-9a31-214b36735532 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:22.938318 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b368a796-a022-47df-9a31-214b36735532 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:22.938624 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b368a796-a022-47df-9a31-214b36735532 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:22.938898 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b368a796-a022-47df-9a31-214b36735532 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:22.948326 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04280d46-822d-402b-907c-d121c68e8b3f None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:22.950368 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b368a796-a022-47df-9a31-214b36735532 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 20 08:47:22.959485 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-b368a796-a022-47df-9a31-214b36735532 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 20 08:47:22.959485 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:47:22.959485 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:47:22.959485 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:47:22.959485 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:47:22.959485 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:47:22.959485 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:47:22.959485 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:47:22.959485 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:47:22.959485 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:47:22.959485 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:47:22.959485 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:47:22.959485 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 20 08:47:22.959485 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 20 08:47:22.959485 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 20 08:47:22.959485 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 20 08:47:22.960417 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 20 08:47:22.960417 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 20 08:47:22.960417 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 20 08:47:22.960417 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 20 08:47:22.960417 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:47:22.960417 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:47:22.960417 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:47:22.960417 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:47:22.960417 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:47:22.960417 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:47:22.960417 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:47:22.960417 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:47:22.960417 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:47:22.960417 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:47:22.960417 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:47:22.960417 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:47:22.961253 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:47:22.961253 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:47:22.961253 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:47:22.961253 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:47:22.961253 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 20 08:47:22.961253 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 20 08:47:22.961253 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 20 08:47:22.961253 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 20 08:47:22.961253 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 20 08:47:22.961253 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 20 08:47:22.961253 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 20 08:47:22.961253 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:47:22.961253 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 7/14] 10.222.0.36 () {56 vars in 1096 bytes} [Thu Oct 20 08:47:22 2022] GET /identity/v3/roles/admin => generated 83 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:47:22.966181 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b368a796-a022-47df-9a31-214b36735532 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:23.046538 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:23.051006 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04280d46-822d-402b-907c-d121c68e8b3f None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:23.053959 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:23.065994 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b368a796-a022-47df-9a31-214b36735532 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:23.089817 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7f6ca94-7782-43de-a9d5-bd31f792c922 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:23.090704 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7f6ca94-7782-43de-a9d5-bd31f792c922 None admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:23.090987 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7f6ca94-7782-43de-a9d5-bd31f792c922 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:23.091258 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7f6ca94-7782-43de-a9d5-bd31f792c922 None admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:23.092231 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7f6ca94-7782-43de-a9d5-bd31f792c922 None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:23.093696 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7f6ca94-7782-43de-a9d5-bd31f792c922 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:23.099443 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b682761-0f92-4016-b7bd-a730d1889b1f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:23.100035 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b682761-0f92-4016-b7bd-a730d1889b1f None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:23.100304 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b682761-0f92-4016-b7bd-a730d1889b1f None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:23.100565 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b682761-0f92-4016-b7bd-a730d1889b1f None admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:23.101219 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b682761-0f92-4016-b7bd-a730d1889b1f None admin] RBAC: Adding query filter params (name=admin) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:47:23.101511 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b682761-0f92-4016-b7bd-a730d1889b1f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:23.102919 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b682761-0f92-4016-b7bd-a730d1889b1f None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:23.105766 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.sql.core [None req-c7f6ca94-7782-43de-a9d5-bd31f792c922 None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 20 08:47:23.105766 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 20 08:47:23.105766 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [parameters: {'id': '779113130ded48f1a0da2720d641590b', 'name': 'Default', 'domain_id': '<>', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': None, 'is_domain': 1}] Oct 20 08:47:23.105766 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81562) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 20 08:47:23.115409 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 8/15] 10.222.0.36 () {56 vars in 1099 bytes} [Thu Oct 20 08:47:22 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 08:47:23.120870 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b682761-0f92-4016-b7bd-a730d1889b1f None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:23.124956 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-c7f6ca94-7782-43de-a9d5-bd31f792c922 None admin] Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default. Oct 20 08:47:23.124956 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:47:23.124956 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 20 08:47:23.124956 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 20 08:47:23.124956 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 20 08:47:23.124956 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 20 08:47:23.124956 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 20 08:47:23.124956 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application result = self._query(query) Oct 20 08:47:23.124956 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 20 08:47:23.124956 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application conn.query(q) Oct 20 08:47:23.124956 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 20 08:47:23.124956 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 20 08:47:23.124956 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 20 08:47:23.124956 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application result.read() Oct 20 08:47:23.125962 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 20 08:47:23.125962 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 20 08:47:23.125962 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 20 08:47:23.125962 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 20 08:47:23.125962 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 20 08:47:23.125962 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 20 08:47:23.125962 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 20 08:47:23.125962 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 20 08:47:23.125962 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 20 08:47:23.125962 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:47:23.125962 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 20 08:47:23.125962 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:47:23.125962 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:47:23.125962 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 20 08:47:23.125962 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 20 08:47:23.125962 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Oct 20 08:47:23.125962 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return project_ref.to_dict() Oct 20 08:47:23.125962 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 20 08:47:23.126985 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application next(self.gen) Oct 20 08:47:23.126985 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Oct 20 08:47:23.126985 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application yield resource Oct 20 08:47:23.126985 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 20 08:47:23.126985 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application next(self.gen) Oct 20 08:47:23.126985 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Oct 20 08:47:23.126985 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application self.session.rollback() Oct 20 08:47:23.126985 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 20 08:47:23.126985 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application self.force_reraise() Oct 20 08:47:23.126985 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 20 08:47:23.126985 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise self.value Oct 20 08:47:23.126985 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Oct 20 08:47:23.126985 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 20 08:47:23.126985 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Oct 20 08:47:23.126985 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application session.commit() Oct 20 08:47:23.126985 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 20 08:47:23.126985 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 20 08:47:23.127852 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 20 08:47:23.127852 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application self._prepare_impl() Oct 20 08:47:23.127852 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 20 08:47:23.127852 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application self.session.flush() Oct 20 08:47:23.127852 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 20 08:47:23.127852 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application self._flush(objects) Oct 20 08:47:23.127852 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 20 08:47:23.127852 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 20 08:47:23.127852 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 20 08:47:23.127852 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application compat.raise_( Oct 20 08:47:23.127852 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 20 08:47:23.127852 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception Oct 20 08:47:23.127852 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 20 08:47:23.127852 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application flush_context.execute() Oct 20 08:47:23.127852 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 20 08:47:23.127852 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rec.execute(self) Oct 20 08:47:23.127852 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 20 08:47:23.128731 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 20 08:47:23.128731 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 20 08:47:23.128731 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 20 08:47:23.128731 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Oct 20 08:47:23.128731 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application c = connection._execute_20( Oct 20 08:47:23.128731 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 20 08:47:23.128731 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 20 08:47:23.128731 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 20 08:47:23.128731 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 20 08:47:23.128731 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 20 08:47:23.128731 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 20 08:47:23.128731 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 20 08:47:23.128731 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 20 08:47:23.128731 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 20 08:47:23.128731 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 20 08:47:23.129486 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 20 08:47:23.129486 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception Oct 20 08:47:23.129486 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 20 08:47:23.129486 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 20 08:47:23.129486 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 20 08:47:23.129486 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 20 08:47:23.129486 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 20 08:47:23.129486 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application result = self._query(query) Oct 20 08:47:23.129486 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 20 08:47:23.129486 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application conn.query(q) Oct 20 08:47:23.129486 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 20 08:47:23.129486 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 20 08:47:23.129486 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 20 08:47:23.129486 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application result.read() Oct 20 08:47:23.129486 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 20 08:47:23.129486 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 20 08:47:23.129486 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 20 08:47:23.131025 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 20 08:47:23.131025 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 20 08:47:23.131025 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 20 08:47:23.131025 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 20 08:47:23.131025 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 20 08:47:23.131025 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 20 08:47:23.131025 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 20 08:47:23.131025 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [parameters: {'id': '779113130ded48f1a0da2720d641590b', 'name': 'Default', 'domain_id': '<>', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': None, 'is_domain': 1}] Oct 20 08:47:23.131025 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 20 08:47:23.131025 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:47:23.131025 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 20 08:47:23.131025 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:47:23.131025 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:47:23.131025 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Oct 20 08:47:23.131025 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Oct 20 08:47:23.132163 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 20 08:47:23.132163 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 20 08:47:23.132163 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name Default. Oct 20 08:47:23.132163 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:47:23.132163 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 20 08:47:23.132163 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:47:23.132163 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:47:23.132163 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:47:23.132163 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:47:23.132163 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:47:23.132163 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:47:23.132163 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:47:23.132163 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:47:23.132163 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:47:23.132163 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:47:23.132163 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:47:23.132163 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:47:23.133290 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 128, in post Oct 20 08:47:23.133290 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_domain( Oct 20 08:47:23.133290 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:47:23.133290 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:47:23.133290 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 765, in create_domain Oct 20 08:47:23.133290 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application is_domain_project = self.create_project( Oct 20 08:47:23.133290 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:47:23.133290 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:47:23.133290 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Oct 20 08:47:23.133290 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.Conflict( Oct 20 08:47:23.133290 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default. Oct 20 08:47:23.133290 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:47:23.133290 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 8/16] 10.222.0.36 () {60 vars in 1137 bytes} [Thu Oct 20 08:47:22 2022] POST /identity/v3/domains => generated 190 bytes in 178 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Oct 20 08:47:23.133290 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7f6ca94-7782-43de-a9d5-bd31f792c922 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:23.225076 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b682761-0f92-4016-b7bd-a730d1889b1f None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:23.231207 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7f6ca94-7782-43de-a9d5-bd31f792c922 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:23.258586 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5077749a-a37d-49fd-8b21-74e9fa982764 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:23.259209 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5077749a-a37d-49fd-8b21-74e9fa982764 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:23.259479 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5077749a-a37d-49fd-8b21-74e9fa982764 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:23.259742 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5077749a-a37d-49fd-8b21-74e9fa982764 None admin] PATH_INFO: `/v3/users/9804008bc9774f7b96bc93c72ddebd2f` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:23.260863 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5077749a-a37d-49fd-8b21-74e9fa982764 None admin] RBAC: Authorizing `identity:get_user(user_id=9804008bc9774f7b96bc93c72ddebd2f)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:23.262300 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5077749a-a37d-49fd-8b21-74e9fa982764 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:23.264026 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 9/17] 10.222.0.36 () {56 vars in 1177 bytes} [Thu Oct 20 08:47:23 2022] GET /identity/v3/users/9804008bc9774f7b96bc93c72ddebd2f => generated 247 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 08:47:23.264865 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b73447d3-1816-445d-9489-d56342da9b67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:23.265715 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b73447d3-1816-445d-9489-d56342da9b67 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:23.265995 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b73447d3-1816-445d-9489-d56342da9b67 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:23.266268 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b73447d3-1816-445d-9489-d56342da9b67 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:23.269201 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5077749a-a37d-49fd-8b21-74e9fa982764 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:23.284033 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:23.285070 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b73447d3-1816-445d-9489-d56342da9b67 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:23.286912 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b73447d3-1816-445d-9489-d56342da9b67 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:23.303003 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-b73447d3-1816-445d-9489-d56342da9b67 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:47:23.303003 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:47:23.303003 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:47:23.303003 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:47:23.303003 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:47:23.303003 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:47:23.303003 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:47:23.303003 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:47:23.303003 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:47:23.303003 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:47:23.303003 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:47:23.303003 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:47:23.303003 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 20 08:47:23.303003 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 20 08:47:23.303003 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 20 08:47:23.303003 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 20 08:47:23.303929 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:47:23.303929 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:47:23.303929 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:47:23.303929 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:47:23.303929 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:47:23.303929 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:47:23.303929 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:47:23.303929 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:47:23.303929 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:47:23.303929 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:47:23.303929 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:47:23.303929 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:47:23.303929 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:47:23.303929 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:47:23.303929 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:47:23.303929 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:47:23.303929 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 20 08:47:23.304761 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 20 08:47:23.304761 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:47:23.304761 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:47:23.304761 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 9/18] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:47:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 20 08:47:23.309876 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b73447d3-1816-445d-9489-d56342da9b67 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:23.331520 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:23.375160 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5077749a-a37d-49fd-8b21-74e9fa982764 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:23.409306 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b73447d3-1816-445d-9489-d56342da9b67 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:23.410376 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce2b056d-9c5e-4439-948e-1a6babfad6ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:23.411036 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce2b056d-9c5e-4439-948e-1a6babfad6ac None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:23.411338 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce2b056d-9c5e-4439-948e-1a6babfad6ac None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:23.411614 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce2b056d-9c5e-4439-948e-1a6babfad6ac None admin] PATH_INFO: `/v3/domains/default` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:23.412715 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce2b056d-9c5e-4439-948e-1a6babfad6ac None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:23.414151 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce2b056d-9c5e-4439-948e-1a6babfad6ac None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:23.415720 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 10/19] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:47:23 2022] GET /identity/v3/domains/default => generated 199 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 08:47:23.421275 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce2b056d-9c5e-4439-948e-1a6babfad6ac None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:23.446337 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-037e9d2c-d63a-4cc7-8348-5e7bdc315a1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:23.446950 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-037e9d2c-d63a-4cc7-8348-5e7bdc315a1e None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:23.447226 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-037e9d2c-d63a-4cc7-8348-5e7bdc315a1e None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:23.447495 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-037e9d2c-d63a-4cc7-8348-5e7bdc315a1e None admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:23.448131 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-037e9d2c-d63a-4cc7-8348-5e7bdc315a1e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:47:23.448432 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-037e9d2c-d63a-4cc7-8348-5e7bdc315a1e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:23.449796 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-037e9d2c-d63a-4cc7-8348-5e7bdc315a1e None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:23.466803 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 10/20] 10.222.0.36 () {56 vars in 1109 bytes} [Thu Oct 20 08:47:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 08:47:23.507054 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:23.527556 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce2b056d-9c5e-4439-948e-1a6babfad6ac None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:23.563015 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e1102cf-f1f3-452c-88e2-c651f4997ab3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:23.563691 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1102cf-f1f3-452c-88e2-c651f4997ab3 None admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:23.563969 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1102cf-f1f3-452c-88e2-c651f4997ab3 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:23.564240 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1102cf-f1f3-452c-88e2-c651f4997ab3 None admin] PATH_INFO: `/v3/domains/default/users/9804008bc9774f7b96bc93c72ddebd2f/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:23.566235 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e1102cf-f1f3-452c-88e2-c651f4997ab3 None admin] RBAC: Authorizing `identity:create_grant(domain_id=default, user_id=9804008bc9774f7b96bc93c72ddebd2f, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:23.567800 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e1102cf-f1f3-452c-88e2-c651f4997ab3 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:23.683064 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 11/21] 10.222.0.36 () {58 vars in 1361 bytes} [Thu Oct 20 08:47:23 2022] PUT /identity/v3/domains/default/users/9804008bc9774f7b96bc93c72ddebd2f/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 262 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 08:47:26.054246 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27abde69-ceff-4586-bdd9-4f0164a03049 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:26.054798 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27abde69-ceff-4586-bdd9-4f0164a03049 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:26.054798 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27abde69-ceff-4586-bdd9-4f0164a03049 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:26.056344 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 11/22] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:47:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:47:26.062723 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46957032-0748-4ebc-9824-e102284eda28 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:26.063186 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46957032-0748-4ebc-9824-e102284eda28 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:26.063283 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46957032-0748-4ebc-9824-e102284eda28 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:26.092249 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-46957032-0748-4ebc-9824-e102284eda28 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:47:26.093791 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-46957032-0748-4ebc-9824-e102284eda28 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:47:26.184670 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 12/23] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:47:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 20 08:47:26.578328 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5052c9d8-808c-402e-836f-2800b0e06c4d None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:26.579064 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5052c9d8-808c-402e-836f-2800b0e06c4d None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:26.579542 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5052c9d8-808c-402e-836f-2800b0e06c4d None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:26.581829 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 12/24] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:47:26 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:47:26.587617 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2db1cf3-f12d-47cc-bcab-1159c1bfe5cc None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:26.588236 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2db1cf3-f12d-47cc-bcab-1159c1bfe5cc None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:26.588723 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2db1cf3-f12d-47cc-bcab-1159c1bfe5cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:26.612533 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-b2db1cf3-f12d-47cc-bcab-1159c1bfe5cc None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:47:26.614087 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-b2db1cf3-f12d-47cc-bcab-1159c1bfe5cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:47:26.635098 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5052c9d8-808c-402e-836f-2800b0e06c4d None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:26.722288 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:26.741916 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 13/25] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:47:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 20 08:47:26.788850 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17bb616c-c6fe-4842-8e1a-bcc4cf3ae1e9 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:26.789155 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17bb616c-c6fe-4842-8e1a-bcc4cf3ae1e9 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:26.789409 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17bb616c-c6fe-4842-8e1a-bcc4cf3ae1e9 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:26.793705 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 14/26] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:47:26 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 20 08:47:26.799982 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b09f31-cc80-45a4-8497-4d6ad5578aaa None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:26.800263 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b09f31-cc80-45a4-8497-4d6ad5578aaa None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:26.800537 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b09f31-cc80-45a4-8497-4d6ad5578aaa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:26.827874 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-73b09f31-cc80-45a4-8497-4d6ad5578aaa None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:47:26.829077 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-73b09f31-cc80-45a4-8497-4d6ad5578aaa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:47:26.832377 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:26.880728 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5052c9d8-808c-402e-836f-2800b0e06c4d None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:26.952616 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 15/27] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:47:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 20 08:47:26.955755 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04edf2d5-1093-4a18-834c-54929aaf665b None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:26.957586 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04edf2d5-1093-4a18-834c-54929aaf665b None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:26.960139 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04edf2d5-1093-4a18-834c-54929aaf665b None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:26.963909 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b332424-4202-4d55-9f2d-03303089356a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:26.964167 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 16/28] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:47:26 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:47:26.965017 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b332424-4202-4d55-9f2d-03303089356a None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:26.965547 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b332424-4202-4d55-9f2d-03303089356a None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:26.966008 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b332424-4202-4d55-9f2d-03303089356a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:26.967637 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46cefd34-0e52-406e-b63f-35b1f0b3738b None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:26.968113 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46cefd34-0e52-406e-b63f-35b1f0b3738b None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:26.968646 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46cefd34-0e52-406e-b63f-35b1f0b3738b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:26.983381 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b332424-4202-4d55-9f2d-03303089356a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:26.985514 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b332424-4202-4d55-9f2d-03303089356a None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:27.002324 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-4b332424-4202-4d55-9f2d-03303089356a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:47:27.002324 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:47:27.002324 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:47:27.002324 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:47:27.002324 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:47:27.002324 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:47:27.002324 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:47:27.002324 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:47:27.002324 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:47:27.002324 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:47:27.002324 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:47:27.002324 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:47:27.002324 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 20 08:47:27.002324 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 20 08:47:27.002324 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 20 08:47:27.002324 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 20 08:47:27.003283 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:47:27.003283 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:47:27.003283 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:47:27.003283 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:47:27.003283 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:47:27.003283 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:47:27.003283 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:47:27.003283 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:47:27.003283 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:47:27.003283 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:47:27.003283 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:47:27.003283 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:47:27.003283 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:47:27.003283 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:47:27.003283 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:47:27.003283 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:47:27.003283 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 20 08:47:27.004098 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 20 08:47:27.004098 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:47:27.004098 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:47:27.006665 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 13/29] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:47:26 2022] GET /identity/v3/domains/Default => generated 87 bytes in 372 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:47:27.012582 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b332424-4202-4d55-9f2d-03303089356a None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:27.014601 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-46cefd34-0e52-406e-b63f-35b1f0b3738b None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:47:27.015883 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-46cefd34-0e52-406e-b63f-35b1f0b3738b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:47:27.131347 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b332424-4202-4d55-9f2d-03303089356a None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:27.133368 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 17/30] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:47:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 20 08:47:27.134704 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46cefd34-0e52-406e-b63f-35b1f0b3738b None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:27.174867 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:27.179599 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0984a4a3-7673-484f-a020-e5dde3aebbd7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:27.180219 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0984a4a3-7673-484f-a020-e5dde3aebbd7 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:27.180495 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0984a4a3-7673-484f-a020-e5dde3aebbd7 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:27.180763 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0984a4a3-7673-484f-a020-e5dde3aebbd7 None admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:27.181410 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0984a4a3-7673-484f-a020-e5dde3aebbd7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:47:27.181707 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0984a4a3-7673-484f-a020-e5dde3aebbd7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:27.184366 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0984a4a3-7673-484f-a020-e5dde3aebbd7 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:27.206152 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 14/31] 10.222.0.36 () {56 vars in 1109 bytes} [Thu Oct 20 08:47:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 08:47:27.212445 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0984a4a3-7673-484f-a020-e5dde3aebbd7 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:27.246354 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:27.332018 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46cefd34-0e52-406e-b63f-35b1f0b3738b None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:27.361548 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0984a4a3-7673-484f-a020-e5dde3aebbd7 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:27.389512 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6fc458b-a6ed-4d9c-ba01-4381f2ae6bef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:27.390424 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6fc458b-a6ed-4d9c-ba01-4381f2ae6bef None admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:27.391016 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6fc458b-a6ed-4d9c-ba01-4381f2ae6bef None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:27.391481 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6fc458b-a6ed-4d9c-ba01-4381f2ae6bef None admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:27.392359 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6fc458b-a6ed-4d9c-ba01-4381f2ae6bef None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:27.394068 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6fc458b-a6ed-4d9c-ba01-4381f2ae6bef None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:27.401260 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c66c02-1d15-48d0-89a1-2ae42bf05d03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:27.401879 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c66c02-1d15-48d0-89a1-2ae42bf05d03 None admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:27.402271 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c66c02-1d15-48d0-89a1-2ae42bf05d03 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:27.402691 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c66c02-1d15-48d0-89a1-2ae42bf05d03 None admin] PATH_INFO: `/v3/projects` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:27.403414 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56c66c02-1d15-48d0-89a1-2ae42bf05d03 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:27.405343 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56c66c02-1d15-48d0-89a1-2ae42bf05d03 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:27.415627 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 18/32] 10.222.0.36 () {60 vars in 1131 bytes} [Thu Oct 20 08:47:27 2022] POST /identity/v3/roles => generated 225 bytes in 281 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 08:47:27.416685 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6fc458b-a6ed-4d9c-ba01-4381f2ae6bef None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:27.578774 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6fc458b-a6ed-4d9c-ba01-4381f2ae6bef None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:27.608096 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:27.612761 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 15/33] 10.222.0.36 () {60 vars in 1141 bytes} [Thu Oct 20 08:47:27 2022] POST /identity/v3/projects => generated 301 bytes in 400 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 20 08:47:27.614347 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb0d420-d59f-4339-82ae-28f0ad6cd4f5 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:27.614700 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb0d420-d59f-4339-82ae-28f0ad6cd4f5 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:27.614984 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb0d420-d59f-4339-82ae-28f0ad6cd4f5 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:27.616371 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 16/34] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:47:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:47:27.617146 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bb0d420-d59f-4339-82ae-28f0ad6cd4f5 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:27.638897 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78e833ae-a17a-465e-ad59-5ffe08f49919 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:27.639536 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e833ae-a17a-465e-ad59-5ffe08f49919 None admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:27.639822 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e833ae-a17a-465e-ad59-5ffe08f49919 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:27.640148 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e833ae-a17a-465e-ad59-5ffe08f49919 None admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:27.640954 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78e833ae-a17a-465e-ad59-5ffe08f49919 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:27.642508 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78e833ae-a17a-465e-ad59-5ffe08f49919 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:27.663708 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 19/35] 10.222.0.36 () {60 vars in 1131 bytes} [Thu Oct 20 08:47:27 2022] POST /identity/v3/roles => generated 219 bytes in 247 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 08:47:27.672251 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8274b823-49f3-41e1-8f86-94a40aebd3c9 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:27.674807 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8274b823-49f3-41e1-8f86-94a40aebd3c9 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:27.675494 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8274b823-49f3-41e1-8f86-94a40aebd3c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:27.698558 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:27.718672 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-8274b823-49f3-41e1-8f86-94a40aebd3c9 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:47:27.720402 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-8274b823-49f3-41e1-8f86-94a40aebd3c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:47:27.800961 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bb0d420-d59f-4339-82ae-28f0ad6cd4f5 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:27.808332 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 20/36] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:47:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 20 08:47:27.809966 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c40198-c59e-4391-b177-5c477b2b05c3 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:27.810402 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c40198-c59e-4391-b177-5c477b2b05c3 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:27.810873 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c40198-c59e-4391-b177-5c477b2b05c3 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:27.812502 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 21/37] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:47:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:47:27.819235 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe49f8e-1b83-4405-98e2-ee97d1224e75 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:27.819835 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe49f8e-1b83-4405-98e2-ee97d1224e75 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:27.820253 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe49f8e-1b83-4405-98e2-ee97d1224e75 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:27.842555 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7a8508a-a089-4d79-8632-7d98ef3ea903 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:27.843157 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a8508a-a089-4d79-8632-7d98ef3ea903 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:27.843438 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a8508a-a089-4d79-8632-7d98ef3ea903 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:27.843710 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a8508a-a089-4d79-8632-7d98ef3ea903 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:27.843902 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-abe49f8e-1b83-4405-98e2-ee97d1224e75 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:47:27.845401 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-abe49f8e-1b83-4405-98e2-ee97d1224e75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:47:27.858768 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7a8508a-a089-4d79-8632-7d98ef3ea903 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 20 08:47:27.867132 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-c7a8508a-a089-4d79-8632-7d98ef3ea903 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 20 08:47:27.867132 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:47:27.867132 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:47:27.867132 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:47:27.867132 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:47:27.867132 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:47:27.867132 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:47:27.867132 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:47:27.867132 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:47:27.867132 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:47:27.867132 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:47:27.867132 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:47:27.867132 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 20 08:47:27.867132 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 20 08:47:27.867132 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 20 08:47:27.867132 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 20 08:47:27.868092 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 20 08:47:27.868092 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 20 08:47:27.868092 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 20 08:47:27.868092 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 20 08:47:27.868092 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:47:27.868092 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:47:27.868092 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:47:27.868092 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:47:27.868092 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:47:27.868092 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:47:27.868092 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:47:27.868092 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:47:27.868092 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:47:27.868092 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:47:27.868092 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:47:27.868092 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:47:27.868864 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:47:27.868864 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:47:27.868864 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:47:27.868864 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:47:27.868864 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 20 08:47:27.868864 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 20 08:47:27.868864 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 20 08:47:27.868864 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 20 08:47:27.868864 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 20 08:47:27.868864 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 20 08:47:27.868864 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 20 08:47:27.868864 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:47:27.868864 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 17/38] 10.222.0.36 () {56 vars in 1096 bytes} [Thu Oct 20 08:47:27 2022] GET /identity/v3/roles/admin => generated 83 bytes in 251 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:47:27.875027 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7a8508a-a089-4d79-8632-7d98ef3ea903 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:28.002696 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7a8508a-a089-4d79-8632-7d98ef3ea903 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:28.006836 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:28.046429 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a65f083-ca41-45d2-9ea5-2a361084bab0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:28.047066 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a65f083-ca41-45d2-9ea5-2a361084bab0 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:28.047346 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a65f083-ca41-45d2-9ea5-2a361084bab0 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:28.047618 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a65f083-ca41-45d2-9ea5-2a361084bab0 None admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:28.048285 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a65f083-ca41-45d2-9ea5-2a361084bab0 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:47:28.048586 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a65f083-ca41-45d2-9ea5-2a361084bab0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:28.050685 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a65f083-ca41-45d2-9ea5-2a361084bab0 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:28.051300 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 22/39] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:47:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 234 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 20 08:47:28.063538 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 18/40] 10.222.0.36 () {56 vars in 1099 bytes} [Thu Oct 20 08:47:27 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:47:28.069922 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abe49f8e-1b83-4405-98e2-ee97d1224e75 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:28.112390 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:28.123035 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a65f083-ca41-45d2-9ea5-2a361084bab0 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:28.183156 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abe49f8e-1b83-4405-98e2-ee97d1224e75 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:28.221564 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e1ef6a2-c794-4bcf-8d7f-1281ad5a8a0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:28.222560 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e1ef6a2-c794-4bcf-8d7f-1281ad5a8a0e None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:28.222999 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e1ef6a2-c794-4bcf-8d7f-1281ad5a8a0e None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:28.223276 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e1ef6a2-c794-4bcf-8d7f-1281ad5a8a0e None admin] PATH_INFO: `/v3/users/9804008bc9774f7b96bc93c72ddebd2f` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:28.224420 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e1ef6a2-c794-4bcf-8d7f-1281ad5a8a0e None admin] RBAC: Authorizing `identity:get_user(user_id=9804008bc9774f7b96bc93c72ddebd2f)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:28.224924 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a65f083-ca41-45d2-9ea5-2a361084bab0 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:28.225871 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e1ef6a2-c794-4bcf-8d7f-1281ad5a8a0e None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:28.227710 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 23/41] 10.222.0.36 () {56 vars in 1177 bytes} [Thu Oct 20 08:47:28 2022] GET /identity/v3/users/9804008bc9774f7b96bc93c72ddebd2f => generated 247 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:47:28.232997 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e1ef6a2-c794-4bcf-8d7f-1281ad5a8a0e None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:28.265182 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e8a47b-94be-4068-9c7c-b33f128cd7b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:28.265911 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e8a47b-94be-4068-9c7c-b33f128cd7b4 None admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:28.266192 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e8a47b-94be-4068-9c7c-b33f128cd7b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:28.266464 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e8a47b-94be-4068-9c7c-b33f128cd7b4 None admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:28.267153 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08e8a47b-94be-4068-9c7c-b33f128cd7b4 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:28.268582 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08e8a47b-94be-4068-9c7c-b33f128cd7b4 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:28.285356 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 19/42] 10.222.0.36 () {60 vars in 1131 bytes} [Thu Oct 20 08:47:28 2022] POST /identity/v3/roles => generated 223 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 20 08:47:28.328061 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e8a47b-94be-4068-9c7c-b33f128cd7b4 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:28.344782 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e1ef6a2-c794-4bcf-8d7f-1281ad5a8a0e None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:28.355853 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:28.366325 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:28.387636 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81ddd431-7048-4713-ac62-d6646936a94f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:28.388348 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ddd431-7048-4713-ac62-d6646936a94f None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:28.388634 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ddd431-7048-4713-ac62-d6646936a94f None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:28.388910 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ddd431-7048-4713-ac62-d6646936a94f None admin] PATH_INFO: `/v3/domains/default` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:28.390085 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81ddd431-7048-4713-ac62-d6646936a94f None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:28.391653 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81ddd431-7048-4713-ac62-d6646936a94f None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:28.393323 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 24/43] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:47:28 2022] GET /identity/v3/domains/default => generated 199 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 08:47:28.400017 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81ddd431-7048-4713-ac62-d6646936a94f None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:28.460472 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e8a47b-94be-4068-9c7c-b33f128cd7b4 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:28.500757 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-673ac618-0694-4073-a6da-a37c3fcc90da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:28.501394 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673ac618-0694-4073-a6da-a37c3fcc90da None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:28.501670 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673ac618-0694-4073-a6da-a37c3fcc90da None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:28.501937 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673ac618-0694-4073-a6da-a37c3fcc90da None admin] PATH_INFO: `/v3/domains/default` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:28.503129 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-673ac618-0694-4073-a6da-a37c3fcc90da None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:28.504928 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-673ac618-0694-4073-a6da-a37c3fcc90da None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:28.506330 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 20/44] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:47:28 2022] GET /identity/v3/domains/default => generated 199 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 08:47:28.511652 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-673ac618-0694-4073-a6da-a37c3fcc90da None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:28.518536 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81ddd431-7048-4713-ac62-d6646936a94f None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:28.554577 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4c29706-d342-41a5-92dd-7ccc651ee1f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:28.555190 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c29706-d342-41a5-92dd-7ccc651ee1f5 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:28.555469 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c29706-d342-41a5-92dd-7ccc651ee1f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:28.555743 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c29706-d342-41a5-92dd-7ccc651ee1f5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:28.556471 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4c29706-d342-41a5-92dd-7ccc651ee1f5 None admin] RBAC: Adding query filter params (role.id=d9d3bc62f6e8453488c74856ef333bb3, scope.domain.id=default, user.id=9804008bc9774f7b96bc93c72ddebd2f) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:47:28.556775 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4c29706-d342-41a5-92dd-7ccc651ee1f5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:28.558230 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4c29706-d342-41a5-92dd-7ccc651ee1f5 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:28.565808 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 25/45] 10.222.0.36 () {56 vars in 1322 bytes} [Thu Oct 20 08:47:28 2022] GET /identity/v3/role_assignments?user.id=9804008bc9774f7b96bc93c72ddebd2f&scope.domain.id=default&role.id=d9d3bc62f6e8453488c74856ef333bb3 => generated 530 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:47:28.608347 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:28.615235 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-673ac618-0694-4073-a6da-a37c3fcc90da None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:28.650680 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6c987d1-406b-4370-a5a0-857c7b678b5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:28.651203 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c987d1-406b-4370-a5a0-857c7b678b5d None admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:28.651482 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c987d1-406b-4370-a5a0-857c7b678b5d None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:28.651750 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c987d1-406b-4370-a5a0-857c7b678b5d None admin] PATH_INFO: `/v3/projects` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:28.652403 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6c987d1-406b-4370-a5a0-857c7b678b5d None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:28.653867 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6c987d1-406b-4370-a5a0-857c7b678b5d None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:28.682777 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 21/46] 10.222.0.36 () {60 vars in 1141 bytes} [Thu Oct 20 08:47:28 2022] POST /identity/v3/projects => generated 312 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 20 08:47:30.810376 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b631d3af-7749-4f65-819a-dd5f48c842b2 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:30.810898 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b631d3af-7749-4f65-819a-dd5f48c842b2 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:30.810898 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b631d3af-7749-4f65-819a-dd5f48c842b2 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:30.812225 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 26/47] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:47:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:47:30.817415 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df5e4b4-9f88-4228-8e8b-9a79bb161a1a None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:30.817821 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df5e4b4-9f88-4228-8e8b-9a79bb161a1a None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:30.817969 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df5e4b4-9f88-4228-8e8b-9a79bb161a1a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:30.836710 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-5df5e4b4-9f88-4228-8e8b-9a79bb161a1a None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:47:30.838192 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-5df5e4b4-9f88-4228-8e8b-9a79bb161a1a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:47:30.902558 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 22/48] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:47:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 20 08:47:31.170979 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b631d3af-7749-4f65-819a-dd5f48c842b2 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:31.214452 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:31.272189 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b631d3af-7749-4f65-819a-dd5f48c842b2 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:31.305178 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96153d14-dbcd-477a-bccd-a45dbac90c93 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:31.305791 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96153d14-dbcd-477a-bccd-a45dbac90c93 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:31.306072 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96153d14-dbcd-477a-bccd-a45dbac90c93 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:31.306343 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96153d14-dbcd-477a-bccd-a45dbac90c93 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:31.307463 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96153d14-dbcd-477a-bccd-a45dbac90c93 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:31.308911 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96153d14-dbcd-477a-bccd-a45dbac90c93 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:31.310443 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 27/49] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:47:31 2022] GET /identity/v3/domains/default => generated 199 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:47:31.316040 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5df5e4b4-9f88-4228-8e8b-9a79bb161a1a None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:31.389400 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:31.414758 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5df5e4b4-9f88-4228-8e8b-9a79bb161a1a None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:31.447533 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97a820b1-fefe-4929-bdea-cd85c517c764 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:31.448079 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a820b1-fefe-4929-bdea-cd85c517c764 None admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:31.448358 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a820b1-fefe-4929-bdea-cd85c517c764 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:31.448626 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a820b1-fefe-4929-bdea-cd85c517c764 None admin] PATH_INFO: `/v3/projects` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:31.449275 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97a820b1-fefe-4929-bdea-cd85c517c764 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:31.450754 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97a820b1-fefe-4929-bdea-cd85c517c764 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:31.476308 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 23/50] 10.222.0.36 () {60 vars in 1140 bytes} [Thu Oct 20 08:47:31 2022] POST /identity/v3/projects => generated 298 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 08:47:33.594163 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd9e8460-f179-42dd-b0a2-9790df003441 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:33.594692 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd9e8460-f179-42dd-b0a2-9790df003441 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:33.594775 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd9e8460-f179-42dd-b0a2-9790df003441 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:33.596089 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 28/51] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:47:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:47:33.601241 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c377d64e-5ee8-4d92-b054-3b99967590c2 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:33.601691 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c377d64e-5ee8-4d92-b054-3b99967590c2 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:33.601788 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c377d64e-5ee8-4d92-b054-3b99967590c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:33.619728 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-c377d64e-5ee8-4d92-b054-3b99967590c2 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:47:33.620918 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-c377d64e-5ee8-4d92-b054-3b99967590c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:47:33.684718 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 24/52] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:47:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 20 08:47:33.956276 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd9e8460-f179-42dd-b0a2-9790df003441 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:34.055514 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd9e8460-f179-42dd-b0a2-9790df003441 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:34.064720 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:34.089953 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4899f7ab-e7ce-4fcb-a440-e9feb7b7b8e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:34.090600 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4899f7ab-e7ce-4fcb-a440-e9feb7b7b8e2 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:34.090883 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4899f7ab-e7ce-4fcb-a440-e9feb7b7b8e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:34.091156 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4899f7ab-e7ce-4fcb-a440-e9feb7b7b8e2 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:34.092253 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4899f7ab-e7ce-4fcb-a440-e9feb7b7b8e2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:34.093693 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4899f7ab-e7ce-4fcb-a440-e9feb7b7b8e2 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:34.095282 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 29/53] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:47:33 2022] GET /identity/v3/domains/default => generated 199 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:47:34.100792 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c377d64e-5ee8-4d92-b054-3b99967590c2 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:34.156809 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:34.197382 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c377d64e-5ee8-4d92-b054-3b99967590c2 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:34.332929 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f41c48b-99a9-438d-afef-7e996e30c197 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:34.333457 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f41c48b-99a9-438d-afef-7e996e30c197 None admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:34.333737 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f41c48b-99a9-438d-afef-7e996e30c197 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:34.334007 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f41c48b-99a9-438d-afef-7e996e30c197 None admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:34.334666 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f41c48b-99a9-438d-afef-7e996e30c197 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:34.336074 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f41c48b-99a9-438d-afef-7e996e30c197 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:34.362351 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 25/54] 10.222.0.36 () {60 vars in 1132 bytes} [Thu Oct 20 08:47:34 2022] POST /identity/v3/users => generated 275 bytes in 262 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 08:47:37.447531 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b0aa6d-27d4-4027-bb7f-c304bbaeb066 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:37.448477 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b0aa6d-27d4-4027-bb7f-c304bbaeb066 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:37.448952 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b0aa6d-27d4-4027-bb7f-c304bbaeb066 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:37.450544 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 30/55] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:47:37 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:47:37.463393 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-513f1c23-d35b-4759-a283-d74f5373df22 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:37.463393 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-513f1c23-d35b-4759-a283-d74f5373df22 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:37.463393 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-513f1c23-d35b-4759-a283-d74f5373df22 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:37.488917 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-513f1c23-d35b-4759-a283-d74f5373df22 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:47:37.490978 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-513f1c23-d35b-4759-a283-d74f5373df22 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:47:37.585989 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 26/56] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:47:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 20 08:47:37.708171 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d0e80a-2349-4a2e-b367-3b15b930e08f None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:37.708707 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d0e80a-2349-4a2e-b367-3b15b930e08f None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:37.709126 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d0e80a-2349-4a2e-b367-3b15b930e08f None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:37.710727 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 31/57] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:47:37 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:47:37.715958 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b1f9ba-6510-4c37-bb74-9d252d85e871 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:37.716250 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b1f9ba-6510-4c37-bb74-9d252d85e871 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:37.716526 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b1f9ba-6510-4c37-bb74-9d252d85e871 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:37.743443 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-80b1f9ba-6510-4c37-bb74-9d252d85e871 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:47:37.744682 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-80b1f9ba-6510-4c37-bb74-9d252d85e871 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:47:37.801544 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:37.835053 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 27/58] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:47:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 20 08:47:37.898323 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce01c8a-a4cf-415c-9b2d-daca8a00c44c None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:37.898948 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce01c8a-a4cf-415c-9b2d-daca8a00c44c None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:37.899404 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce01c8a-a4cf-415c-9b2d-daca8a00c44c None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:37.900939 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 32/59] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:47:37 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:47:37.906208 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f231008-0c4c-43f5-ab89-45160c63f3ba None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:37.906511 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f231008-0c4c-43f5-ab89-45160c63f3ba None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:37.906794 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f231008-0c4c-43f5-ab89-45160c63f3ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:37.928392 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-4f231008-0c4c-43f5-ab89-45160c63f3ba None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:47:37.929553 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-4f231008-0c4c-43f5-ab89-45160c63f3ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:47:37.965486 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce01c8a-a4cf-415c-9b2d-daca8a00c44c None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:38.028342 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 28/60] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:47:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 20 08:47:38.101958 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce01c8a-a4cf-415c-9b2d-daca8a00c44c None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:38.136640 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aee9ee9-125a-4bb9-8744-765fb00aabfa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:38.137482 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aee9ee9-125a-4bb9-8744-765fb00aabfa None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:38.137898 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aee9ee9-125a-4bb9-8744-765fb00aabfa None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:38.138329 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aee9ee9-125a-4bb9-8744-765fb00aabfa None admin] PATH_INFO: `/v3/roles/member` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:38.149082 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aee9ee9-125a-4bb9-8744-765fb00aabfa None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 20 08:47:38.154073 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-2aee9ee9-125a-4bb9-8744-765fb00aabfa None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 20 08:47:38.154073 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:47:38.154073 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:47:38.154073 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:47:38.154073 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:47:38.154073 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:47:38.154073 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:47:38.154073 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:47:38.154073 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:47:38.154073 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:47:38.154073 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:47:38.154073 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:47:38.154073 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 20 08:47:38.154073 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 20 08:47:38.154073 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 20 08:47:38.154073 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 20 08:47:38.155116 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 20 08:47:38.155116 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 20 08:47:38.155116 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 20 08:47:38.155116 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 20 08:47:38.155116 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:47:38.155116 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:47:38.155116 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:47:38.155116 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:47:38.155116 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:47:38.155116 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:47:38.155116 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:47:38.155116 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:47:38.155116 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:47:38.155116 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:47:38.155116 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:47:38.155116 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:47:38.156062 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:47:38.156062 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:47:38.156062 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:47:38.156062 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:47:38.156062 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 20 08:47:38.156062 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 20 08:47:38.156062 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 20 08:47:38.156062 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 20 08:47:38.156062 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 20 08:47:38.156062 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 20 08:47:38.156062 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 20 08:47:38.156062 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:47:38.158193 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 33/61] 10.222.0.36 () {56 vars in 1099 bytes} [Thu Oct 20 08:47:37 2022] GET /identity/v3/roles/member => generated 84 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 20 08:47:38.165256 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f231008-0c4c-43f5-ab89-45160c63f3ba None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:38.174064 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aee9ee9-125a-4bb9-8744-765fb00aabfa None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:38.243209 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:38.332130 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aee9ee9-125a-4bb9-8744-765fb00aabfa None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:38.360798 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f231008-0c4c-43f5-ab89-45160c63f3ba None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:38.367668 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:38.379272 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a553de4-0690-41fe-8e09-23ba0f2e51a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:38.380253 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a553de4-0690-41fe-8e09-23ba0f2e51a4 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:38.380684 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a553de4-0690-41fe-8e09-23ba0f2e51a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:38.381105 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a553de4-0690-41fe-8e09-23ba0f2e51a4 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:38.393381 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a553de4-0690-41fe-8e09-23ba0f2e51a4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 20 08:47:38.398843 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-2a553de4-0690-41fe-8e09-23ba0f2e51a4 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 20 08:47:38.398843 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:47:38.398843 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:47:38.398843 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:47:38.398843 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:47:38.398843 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:47:38.398843 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:47:38.398843 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:47:38.398843 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:47:38.398843 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:47:38.398843 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:47:38.398843 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:47:38.398843 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 20 08:47:38.398843 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 20 08:47:38.398843 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 20 08:47:38.398843 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 20 08:47:38.399693 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 20 08:47:38.399693 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 20 08:47:38.399693 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 20 08:47:38.399693 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 20 08:47:38.399693 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:47:38.399693 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:47:38.399693 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:47:38.399693 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:47:38.399693 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:47:38.399693 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:47:38.399693 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:47:38.399693 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:47:38.399693 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:47:38.399693 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:47:38.399693 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:47:38.399693 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:47:38.400462 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:47:38.400462 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:47:38.400462 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:47:38.400462 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:47:38.400462 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 20 08:47:38.400462 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 20 08:47:38.400462 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 20 08:47:38.400462 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 20 08:47:38.400462 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 20 08:47:38.400462 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 20 08:47:38.400462 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 20 08:47:38.400462 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:47:38.400462 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 34/62] 10.222.0.36 () {56 vars in 1096 bytes} [Thu Oct 20 08:47:38 2022] GET /identity/v3/roles/admin => generated 83 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:47:38.401814 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e973797-e769-4172-90c9-b9df56abb47c None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:38.401814 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e973797-e769-4172-90c9-b9df56abb47c None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:38.401814 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e973797-e769-4172-90c9-b9df56abb47c None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:38.403492 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 35/63] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:47:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:47:38.404195 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e973797-e769-4172-90c9-b9df56abb47c None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:38.413883 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38020952-3218-4b65-9a56-a6de1c7b3b5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:38.414557 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38020952-3218-4b65-9a56-a6de1c7b3b5e None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:38.414848 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38020952-3218-4b65-9a56-a6de1c7b3b5e None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:38.415125 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38020952-3218-4b65-9a56-a6de1c7b3b5e None admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:38.415801 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38020952-3218-4b65-9a56-a6de1c7b3b5e None admin] RBAC: Adding query filter params (name=member) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:47:38.416108 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38020952-3218-4b65-9a56-a6de1c7b3b5e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:38.417516 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38020952-3218-4b65-9a56-a6de1c7b3b5e None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:38.430558 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 29/64] 10.222.0.36 () {56 vars in 1101 bytes} [Thu Oct 20 08:47:38 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 262 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:47:38.430558 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38020952-3218-4b65-9a56-a6de1c7b3b5e None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:38.512978 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:38.521499 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e973797-e769-4172-90c9-b9df56abb47c None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:38.537586 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38020952-3218-4b65-9a56-a6de1c7b3b5e None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:38.556797 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c1afe3-2c17-4a54-b609-8445b3af4d84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:38.557437 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c1afe3-2c17-4a54-b609-8445b3af4d84 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:38.557721 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c1afe3-2c17-4a54-b609-8445b3af4d84 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:38.557996 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c1afe3-2c17-4a54-b609-8445b3af4d84 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:38.568442 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2c1afe3-2c17-4a54-b609-8445b3af4d84 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 20 08:47:38.573234 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-c2c1afe3-2c17-4a54-b609-8445b3af4d84 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 20 08:47:38.573234 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:47:38.573234 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:47:38.573234 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:47:38.573234 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:47:38.573234 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:47:38.573234 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:47:38.573234 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:47:38.573234 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:47:38.573234 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:47:38.573234 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:47:38.573234 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:47:38.573234 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 20 08:47:38.573234 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 20 08:47:38.573234 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 20 08:47:38.573234 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 20 08:47:38.574098 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 20 08:47:38.574098 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 20 08:47:38.574098 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 20 08:47:38.574098 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 20 08:47:38.574098 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:47:38.574098 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:47:38.574098 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:47:38.574098 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:47:38.574098 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:47:38.574098 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:47:38.574098 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:47:38.574098 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:47:38.574098 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:47:38.574098 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:47:38.574098 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:47:38.574098 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:47:38.574919 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:47:38.574919 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:47:38.574919 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:47:38.574919 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:47:38.574919 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 20 08:47:38.574919 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 20 08:47:38.574919 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 20 08:47:38.574919 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 20 08:47:38.574919 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 20 08:47:38.574919 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 20 08:47:38.574919 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 20 08:47:38.574919 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:47:38.576870 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22379ec4-29b8-4464-925c-19ecc5d4309e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:38.577504 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22379ec4-29b8-4464-925c-19ecc5d4309e None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:38.577760 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22379ec4-29b8-4464-925c-19ecc5d4309e None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:38.578037 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22379ec4-29b8-4464-925c-19ecc5d4309e None admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:38.578743 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22379ec4-29b8-4464-925c-19ecc5d4309e None admin] RBAC: Adding query filter params (name=admin) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:47:38.579147 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22379ec4-29b8-4464-925c-19ecc5d4309e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:38.579525 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 36/65] 10.222.0.36 () {56 vars in 1114 bytes} [Thu Oct 20 08:47:38 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:47:38.580547 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22379ec4-29b8-4464-925c-19ecc5d4309e None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:38.580637 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36bfe6fe-84fc-48af-9c6b-5f264b66ae74 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:38.580930 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36bfe6fe-84fc-48af-9c6b-5f264b66ae74 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:38.581474 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36bfe6fe-84fc-48af-9c6b-5f264b66ae74 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:38.597363 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 30/66] 10.222.0.36 () {56 vars in 1099 bytes} [Thu Oct 20 08:47:38 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:47:38.597363 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22379ec4-29b8-4464-925c-19ecc5d4309e None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:38.601400 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-36bfe6fe-84fc-48af-9c6b-5f264b66ae74 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:47:38.602627 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-36bfe6fe-84fc-48af-9c6b-5f264b66ae74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:47:38.625209 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:38.675799 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 37/67] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:47:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 20 08:47:38.678704 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36bfe6fe-84fc-48af-9c6b-5f264b66ae74 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:38.731646 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22379ec4-29b8-4464-925c-19ecc5d4309e None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:38.779149 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-740d2943-cd7c-4e31-a72c-cb10e0317629 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:38.779712 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740d2943-cd7c-4e31-a72c-cb10e0317629 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:38.780001 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740d2943-cd7c-4e31-a72c-cb10e0317629 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:38.780275 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740d2943-cd7c-4e31-a72c-cb10e0317629 None admin] PATH_INFO: `/v3/users/3cfc8055c0054f8f837edfcbaf0b0fe3` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:38.795766 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-740d2943-cd7c-4e31-a72c-cb10e0317629 None admin] RBAC: Authorizing `identity:get_user(user_id=3cfc8055c0054f8f837edfcbaf0b0fe3)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:38.797243 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-740d2943-cd7c-4e31-a72c-cb10e0317629 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:38.798908 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 31/68] 10.222.0.36 () {56 vars in 1177 bytes} [Thu Oct 20 08:47:38 2022] GET /identity/v3/users/3cfc8055c0054f8f837edfcbaf0b0fe3 => generated 275 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:47:38.799685 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-740d2943-cd7c-4e31-a72c-cb10e0317629 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:38.800616 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:38.816448 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36bfe6fe-84fc-48af-9c6b-5f264b66ae74 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:38.851853 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22e269c4-71fe-41e5-a03d-5e62c155ca94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:38.852460 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e269c4-71fe-41e5-a03d-5e62c155ca94 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:38.852738 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e269c4-71fe-41e5-a03d-5e62c155ca94 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:38.852998 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e269c4-71fe-41e5-a03d-5e62c155ca94 None admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:38.853712 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22e269c4-71fe-41e5-a03d-5e62c155ca94 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:47:38.854018 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22e269c4-71fe-41e5-a03d-5e62c155ca94 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:38.855418 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22e269c4-71fe-41e5-a03d-5e62c155ca94 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:38.869695 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 38/69] 10.222.0.36 () {56 vars in 1111 bytes} [Thu Oct 20 08:47:38 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:47:38.869695 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd7bb8f-f069-46c8-8ea4-de01f0df6ded None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:38.869695 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd7bb8f-f069-46c8-8ea4-de01f0df6ded None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:38.869695 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd7bb8f-f069-46c8-8ea4-de01f0df6ded None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:38.869695 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 39/70] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:47:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:47:38.869695 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cd7bb8f-f069-46c8-8ea4-de01f0df6ded None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:38.897924 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:38.915551 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-740d2943-cd7c-4e31-a72c-cb10e0317629 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:38.951946 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17863a84-09ce-46e6-9573-c41ae485ceeb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:38.952587 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17863a84-09ce-46e6-9573-c41ae485ceeb None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:38.952871 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17863a84-09ce-46e6-9573-c41ae485ceeb None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:38.953146 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17863a84-09ce-46e6-9573-c41ae485ceeb None admin] PATH_INFO: `/v3/users/9804008bc9774f7b96bc93c72ddebd2f` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:38.954575 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17863a84-09ce-46e6-9573-c41ae485ceeb None admin] RBAC: Authorizing `identity:get_user(user_id=9804008bc9774f7b96bc93c72ddebd2f)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:38.956032 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17863a84-09ce-46e6-9573-c41ae485ceeb None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:38.957875 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 32/71] 10.222.0.36 () {56 vars in 1177 bytes} [Thu Oct 20 08:47:38 2022] GET /identity/v3/users/9804008bc9774f7b96bc93c72ddebd2f => generated 247 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:47:38.959592 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6687132d-45f2-4d1d-8ecb-c24039de1c9d None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:38.959878 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6687132d-45f2-4d1d-8ecb-c24039de1c9d None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:38.960158 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6687132d-45f2-4d1d-8ecb-c24039de1c9d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:38.974814 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cd7bb8f-f069-46c8-8ea4-de01f0df6ded None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:38.977783 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-6687132d-45f2-4d1d-8ecb-c24039de1c9d None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:47:38.979007 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-6687132d-45f2-4d1d-8ecb-c24039de1c9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:47:39.008586 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a62f78fd-6c26-474d-9372-70c139b20ded None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:39.009219 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a62f78fd-6c26-474d-9372-70c139b20ded None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:39.009504 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a62f78fd-6c26-474d-9372-70c139b20ded None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:39.009781 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a62f78fd-6c26-474d-9372-70c139b20ded None admin] PATH_INFO: `/v3/projects/fcebd94752274a34a2068dd59f6fe700` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:39.010946 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a62f78fd-6c26-474d-9372-70c139b20ded None admin] RBAC: Authorizing `identity:get_project(project_id=fcebd94752274a34a2068dd59f6fe700)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:39.012395 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a62f78fd-6c26-474d-9372-70c139b20ded None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:39.014023 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 40/72] 10.222.0.36 () {56 vars in 1186 bytes} [Thu Oct 20 08:47:38 2022] GET /identity/v3/projects/fcebd94752274a34a2068dd59f6fe700 => generated 298 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:47:39.014808 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a62f78fd-6c26-474d-9372-70c139b20ded None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:39.033034 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:39.050864 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 33/73] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:47:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 20 08:47:39.051710 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6687132d-45f2-4d1d-8ecb-c24039de1c9d None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:39.120634 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a62f78fd-6c26-474d-9372-70c139b20ded None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:39.140969 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:39.159302 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abde4ed4-b1ee-4229-ae2d-15e57f7884fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:39.160232 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abde4ed4-b1ee-4229-ae2d-15e57f7884fd None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:39.160358 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6687132d-45f2-4d1d-8ecb-c24039de1c9d None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:39.160831 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abde4ed4-b1ee-4229-ae2d-15e57f7884fd None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:39.161298 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abde4ed4-b1ee-4229-ae2d-15e57f7884fd None admin] PATH_INFO: `/v3/users/3cfc8055c0054f8f837edfcbaf0b0fe3` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:39.162664 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abde4ed4-b1ee-4229-ae2d-15e57f7884fd None admin] RBAC: Authorizing `identity:get_user(user_id=3cfc8055c0054f8f837edfcbaf0b0fe3)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:39.164235 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abde4ed4-b1ee-4229-ae2d-15e57f7884fd None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:39.170294 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 41/74] 10.222.0.36 () {56 vars in 1177 bytes} [Thu Oct 20 08:47:39 2022] GET /identity/v3/users/3cfc8055c0054f8f837edfcbaf0b0fe3 => generated 275 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:47:39.170294 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abde4ed4-b1ee-4229-ae2d-15e57f7884fd None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:39.198042 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f841260-3234-422c-b854-044ea947d5ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:39.198705 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f841260-3234-422c-b854-044ea947d5ce None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:39.199094 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f841260-3234-422c-b854-044ea947d5ce None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:39.199369 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f841260-3234-422c-b854-044ea947d5ce None admin] PATH_INFO: `/v3/roles/member` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:39.210244 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f841260-3234-422c-b854-044ea947d5ce None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 20 08:47:39.215054 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-5f841260-3234-422c-b854-044ea947d5ce None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 20 08:47:39.215054 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:47:39.215054 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:47:39.215054 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:47:39.215054 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:47:39.215054 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:47:39.215054 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:47:39.215054 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:47:39.215054 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:47:39.215054 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:47:39.215054 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:47:39.215054 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:47:39.215054 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 20 08:47:39.215054 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 20 08:47:39.215054 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 20 08:47:39.215054 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 20 08:47:39.215942 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 20 08:47:39.215942 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 20 08:47:39.215942 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 20 08:47:39.215942 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 20 08:47:39.215942 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:47:39.215942 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:47:39.215942 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:47:39.215942 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:47:39.215942 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:47:39.215942 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:47:39.215942 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:47:39.215942 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:47:39.215942 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:47:39.215942 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:47:39.215942 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:47:39.215942 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:47:39.216723 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:47:39.216723 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:47:39.216723 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:47:39.216723 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:47:39.216723 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 20 08:47:39.216723 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 20 08:47:39.216723 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 20 08:47:39.216723 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 20 08:47:39.216723 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 20 08:47:39.216723 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 20 08:47:39.216723 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 20 08:47:39.216723 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:47:39.216723 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 34/75] 10.222.0.36 () {56 vars in 1099 bytes} [Thu Oct 20 08:47:39 2022] GET /identity/v3/roles/member => generated 84 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:47:39.217765 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f841260-3234-422c-b854-044ea947d5ce None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:39.263540 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:39.279793 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abde4ed4-b1ee-4229-ae2d-15e57f7884fd None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:39.316920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf364d8a-7ec9-498f-bb5f-94a6f11d8c2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:39.317478 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf364d8a-7ec9-498f-bb5f-94a6f11d8c2f None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:39.318353 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf364d8a-7ec9-498f-bb5f-94a6f11d8c2f None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:39.318667 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf364d8a-7ec9-498f-bb5f-94a6f11d8c2f None admin] PATH_INFO: `/v3/projects/fcebd94752274a34a2068dd59f6fe700` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:39.319785 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf364d8a-7ec9-498f-bb5f-94a6f11d8c2f None admin] RBAC: Authorizing `identity:get_project(project_id=fcebd94752274a34a2068dd59f6fe700)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:39.321241 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf364d8a-7ec9-498f-bb5f-94a6f11d8c2f None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:39.323276 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 42/76] 10.222.0.36 () {56 vars in 1186 bytes} [Thu Oct 20 08:47:39 2022] GET /identity/v3/projects/fcebd94752274a34a2068dd59f6fe700 => generated 298 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:47:39.324178 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf364d8a-7ec9-498f-bb5f-94a6f11d8c2f None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:39.330727 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f841260-3234-422c-b854-044ea947d5ce None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:39.366884 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1afeadde-8241-4df2-86a6-10202dd8cfff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:39.367827 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1afeadde-8241-4df2-86a6-10202dd8cfff None admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:39.368114 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1afeadde-8241-4df2-86a6-10202dd8cfff None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:39.368387 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1afeadde-8241-4df2-86a6-10202dd8cfff None admin] PATH_INFO: `/v3/projects/fcebd94752274a34a2068dd59f6fe700/users/3cfc8055c0054f8f837edfcbaf0b0fe3/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:39.370358 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1afeadde-8241-4df2-86a6-10202dd8cfff None admin] RBAC: Authorizing `identity:create_grant(project_id=fcebd94752274a34a2068dd59f6fe700, user_id=3cfc8055c0054f8f837edfcbaf0b0fe3, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:39.371862 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1afeadde-8241-4df2-86a6-10202dd8cfff None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:39.388156 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 35/77] 10.222.0.36 () {58 vars in 1439 bytes} [Thu Oct 20 08:47:39 2022] PUT /identity/v3/projects/fcebd94752274a34a2068dd59f6fe700/users/3cfc8055c0054f8f837edfcbaf0b0fe3/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 08:47:39.388974 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1afeadde-8241-4df2-86a6-10202dd8cfff None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:39.393228 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:39.430407 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf364d8a-7ec9-498f-bb5f-94a6f11d8c2f None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:39.467660 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7744a8f-f8c3-4246-be38-caacec07e819 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:39.468472 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7744a8f-f8c3-4246-be38-caacec07e819 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:39.468580 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7744a8f-f8c3-4246-be38-caacec07e819 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:39.468848 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7744a8f-f8c3-4246-be38-caacec07e819 None admin] PATH_INFO: `/v3/projects/fcebd94752274a34a2068dd59f6fe700` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:39.470015 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7744a8f-f8c3-4246-be38-caacec07e819 None admin] RBAC: Authorizing `identity:get_project(project_id=fcebd94752274a34a2068dd59f6fe700)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:39.471536 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7744a8f-f8c3-4246-be38-caacec07e819 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:39.473243 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 43/78] 10.222.0.36 () {56 vars in 1186 bytes} [Thu Oct 20 08:47:39 2022] GET /identity/v3/projects/fcebd94752274a34a2068dd59f6fe700 => generated 298 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:47:39.474684 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7744a8f-f8c3-4246-be38-caacec07e819 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:39.498807 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:39.513058 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1afeadde-8241-4df2-86a6-10202dd8cfff None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:39.551242 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb2482c-8645-4405-b393-9533e90ae05a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:39.551880 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb2482c-8645-4405-b393-9533e90ae05a None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:39.552749 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb2482c-8645-4405-b393-9533e90ae05a None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:39.553031 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb2482c-8645-4405-b393-9533e90ae05a None admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:39.553712 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbb2482c-8645-4405-b393-9533e90ae05a None admin] RBAC: Adding query filter params (name=member) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:47:39.554009 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbb2482c-8645-4405-b393-9533e90ae05a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:39.555446 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbb2482c-8645-4405-b393-9533e90ae05a None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:39.567731 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 36/79] 10.222.0.36 () {56 vars in 1101 bytes} [Thu Oct 20 08:47:39 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:47:39.568401 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb2482c-8645-4405-b393-9533e90ae05a None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:39.587633 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7744a8f-f8c3-4246-be38-caacec07e819 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:39.622345 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e0cdeb-513c-4beb-919f-f2c6a3bb185a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:39.622997 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e0cdeb-513c-4beb-919f-f2c6a3bb185a None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:39.623278 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e0cdeb-513c-4beb-919f-f2c6a3bb185a None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:39.623553 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e0cdeb-513c-4beb-919f-f2c6a3bb185a None admin] PATH_INFO: `/v3/domains/default` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:39.624691 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e0cdeb-513c-4beb-919f-f2c6a3bb185a None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:39.626134 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e0cdeb-513c-4beb-919f-f2c6a3bb185a None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:39.627752 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 44/80] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:47:39 2022] GET /identity/v3/domains/default => generated 199 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:47:39.628580 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e0cdeb-513c-4beb-919f-f2c6a3bb185a None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:39.645064 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:39.688915 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb2482c-8645-4405-b393-9533e90ae05a None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:39.723045 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c4ccb4c-bc06-48fb-9efe-fd7c9551e2f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:39.723702 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c4ccb4c-bc06-48fb-9efe-fd7c9551e2f2 None admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:39.724489 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c4ccb4c-bc06-48fb-9efe-fd7c9551e2f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:39.724768 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c4ccb4c-bc06-48fb-9efe-fd7c9551e2f2 None admin] PATH_INFO: `/v3/projects/fcebd94752274a34a2068dd59f6fe700/users/9804008bc9774f7b96bc93c72ddebd2f/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:39.726771 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c4ccb4c-bc06-48fb-9efe-fd7c9551e2f2 None admin] RBAC: Authorizing `identity:create_grant(project_id=fcebd94752274a34a2068dd59f6fe700, user_id=9804008bc9774f7b96bc93c72ddebd2f, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:39.727433 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:39.728577 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c4ccb4c-bc06-48fb-9efe-fd7c9551e2f2 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:39.742465 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e0cdeb-513c-4beb-919f-f2c6a3bb185a None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:39.743270 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 37/81] 10.222.0.36 () {58 vars in 1439 bytes} [Thu Oct 20 08:47:39 2022] PUT /identity/v3/projects/fcebd94752274a34a2068dd59f6fe700/users/9804008bc9774f7b96bc93c72ddebd2f/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 08:47:39.751483 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c4ccb4c-bc06-48fb-9efe-fd7c9551e2f2 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:39.781660 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55af5dd1-ef10-43d6-9dca-17d89caf38e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:39.782370 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55af5dd1-ef10-43d6-9dca-17d89caf38e6 None admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:39.782713 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55af5dd1-ef10-43d6-9dca-17d89caf38e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:39.782998 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55af5dd1-ef10-43d6-9dca-17d89caf38e6 None admin] PATH_INFO: `/v3/projects/fcebd94752274a34a2068dd59f6fe700/users/3cfc8055c0054f8f837edfcbaf0b0fe3/roles/b3f8d63bf4c4488288e57693c6e9804c` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:39.785784 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55af5dd1-ef10-43d6-9dca-17d89caf38e6 None admin] RBAC: Authorizing `identity:create_grant(project_id=fcebd94752274a34a2068dd59f6fe700, user_id=3cfc8055c0054f8f837edfcbaf0b0fe3, role_id=b3f8d63bf4c4488288e57693c6e9804c)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:39.787992 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55af5dd1-ef10-43d6-9dca-17d89caf38e6 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:39.806286 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 45/82] 10.222.0.36 () {58 vars in 1439 bytes} [Thu Oct 20 08:47:39 2022] PUT /identity/v3/projects/fcebd94752274a34a2068dd59f6fe700/users/3cfc8055c0054f8f837edfcbaf0b0fe3/roles/b3f8d63bf4c4488288e57693c6e9804c => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 08:47:39.807183 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55af5dd1-ef10-43d6-9dca-17d89caf38e6 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:39.888254 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c4ccb4c-bc06-48fb-9efe-fd7c9551e2f2 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:39.936350 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:39.951331 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54a992a8-1f33-4405-9eec-9ba1e7478c24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:39.952501 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a992a8-1f33-4405-9eec-9ba1e7478c24 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:39.953009 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a992a8-1f33-4405-9eec-9ba1e7478c24 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:39.953664 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a992a8-1f33-4405-9eec-9ba1e7478c24 None admin] PATH_INFO: `/v3/users/3cfc8055c0054f8f837edfcbaf0b0fe3` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:39.955921 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54a992a8-1f33-4405-9eec-9ba1e7478c24 None admin] RBAC: Authorizing `identity:get_user(user_id=3cfc8055c0054f8f837edfcbaf0b0fe3)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:39.957532 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55af5dd1-ef10-43d6-9dca-17d89caf38e6 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:39.958144 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54a992a8-1f33-4405-9eec-9ba1e7478c24 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:39.961342 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 38/83] 10.222.0.36 () {56 vars in 1177 bytes} [Thu Oct 20 08:47:39 2022] GET /identity/v3/users/3cfc8055c0054f8f837edfcbaf0b0fe3 => generated 275 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:47:39.967808 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54a992a8-1f33-4405-9eec-9ba1e7478c24 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:40.002739 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5cb602e-7579-4272-8029-243e606769da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:40.003543 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5cb602e-7579-4272-8029-243e606769da None admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:40.003821 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5cb602e-7579-4272-8029-243e606769da None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:40.004090 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5cb602e-7579-4272-8029-243e606769da None admin] PATH_INFO: `/v3/users` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:40.004769 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5cb602e-7579-4272-8029-243e606769da None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:40.007902 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5cb602e-7579-4272-8029-243e606769da None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:40.047487 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 46/84] 10.222.0.36 () {60 vars in 1132 bytes} [Thu Oct 20 08:47:39 2022] POST /identity/v3/users => generated 289 bytes in 240 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 08:47:40.108752 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54a992a8-1f33-4405-9eec-9ba1e7478c24 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:40.150919 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c243b5a3-d10a-41ac-bed5-a3bf18f32b88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:40.151973 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c243b5a3-d10a-41ac-bed5-a3bf18f32b88 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:40.152460 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c243b5a3-d10a-41ac-bed5-a3bf18f32b88 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:40.152949 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c243b5a3-d10a-41ac-bed5-a3bf18f32b88 None admin] PATH_INFO: `/v3/projects/147a9fa19b3d4014b5dec4a105c0c0f7` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:40.154497 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c243b5a3-d10a-41ac-bed5-a3bf18f32b88 None admin] RBAC: Authorizing `identity:get_project(project_id=147a9fa19b3d4014b5dec4a105c0c0f7)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:40.156207 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c243b5a3-d10a-41ac-bed5-a3bf18f32b88 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:40.158688 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 39/85] 10.222.0.36 () {56 vars in 1186 bytes} [Thu Oct 20 08:47:39 2022] GET /identity/v3/projects/147a9fa19b3d4014b5dec4a105c0c0f7 => generated 312 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:47:40.165409 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5cb602e-7579-4272-8029-243e606769da None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:40.182870 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:40.306118 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5cb602e-7579-4272-8029-243e606769da None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:40.348938 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e9aef03-855e-4f37-a360-1ca809920a30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:40.349649 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e9aef03-855e-4f37-a360-1ca809920a30 None admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:40.349930 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e9aef03-855e-4f37-a360-1ca809920a30 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:40.350200 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e9aef03-855e-4f37-a360-1ca809920a30 None admin] PATH_INFO: `/v3/projects/147a9fa19b3d4014b5dec4a105c0c0f7/users/3cfc8055c0054f8f837edfcbaf0b0fe3/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:40.355866 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e9aef03-855e-4f37-a360-1ca809920a30 None admin] RBAC: Authorizing `identity:create_grant(project_id=147a9fa19b3d4014b5dec4a105c0c0f7, user_id=3cfc8055c0054f8f837edfcbaf0b0fe3, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:40.357465 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e9aef03-855e-4f37-a360-1ca809920a30 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:40.380831 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 47/86] 10.222.0.36 () {58 vars in 1439 bytes} [Thu Oct 20 08:47:40 2022] PUT /identity/v3/projects/147a9fa19b3d4014b5dec4a105c0c0f7/users/3cfc8055c0054f8f837edfcbaf0b0fe3/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 215 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 08:47:41.944745 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59f75c9b-4974-434b-bbb1-f9b9f00e468d None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:41.945259 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59f75c9b-4974-434b-bbb1-f9b9f00e468d None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:41.945334 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59f75c9b-4974-434b-bbb1-f9b9f00e468d None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:41.946953 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 40/87] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:47:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:47:41.952062 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6bd0471-1f88-4fdf-9034-7fe46c327aca None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:41.952486 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6bd0471-1f88-4fdf-9034-7fe46c327aca None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:41.952667 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6bd0471-1f88-4fdf-9034-7fe46c327aca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:41.976869 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-a6bd0471-1f88-4fdf-9034-7fe46c327aca None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:47:41.978073 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-a6bd0471-1f88-4fdf-9034-7fe46c327aca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:47:42.041994 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:42.053585 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 48/88] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:47:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 20 08:47:42.577598 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d47c72e-6a6d-4222-a543-ae516343570f None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:42.577961 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d47c72e-6a6d-4222-a543-ae516343570f None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:42.578158 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d47c72e-6a6d-4222-a543-ae516343570f None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:42.579622 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 41/89] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:47:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:47:42.588119 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6bd0471-1f88-4fdf-9034-7fe46c327aca None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:42.589798 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6158da52-55c4-4899-b3ec-73d477322278 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:42.590083 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6158da52-55c4-4899-b3ec-73d477322278 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:42.590360 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6158da52-55c4-4899-b3ec-73d477322278 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:42.621691 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-6158da52-55c4-4899-b3ec-73d477322278 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:47:42.622933 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-6158da52-55c4-4899-b3ec-73d477322278 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:47:42.688861 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:42.704951 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 42/90] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:47:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (3 switches on core 0) Oct 20 08:47:42.753625 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6bd0471-1f88-4fdf-9034-7fe46c327aca None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:42.782655 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c2153b-ab34-42af-851c-f5364dbd5c6b None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:42.783267 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c2153b-ab34-42af-851c-f5364dbd5c6b None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:42.783736 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c2153b-ab34-42af-851c-f5364dbd5c6b None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:42.785280 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 43/91] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:47:42 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 20 08:47:42.791745 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f29c845-6446-43a8-8713-c6e51898e8b9 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:42.792297 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f29c845-6446-43a8-8713-c6e51898e8b9 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:42.792760 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f29c845-6446-43a8-8713-c6e51898e8b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:42.803173 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb74ac04-58f9-48ea-aa84-5127c8115c8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:42.808805 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb74ac04-58f9-48ea-aa84-5127c8115c8f None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:42.809295 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb74ac04-58f9-48ea-aa84-5127c8115c8f None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:42.809855 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb74ac04-58f9-48ea-aa84-5127c8115c8f None admin] PATH_INFO: `/v3/roles/member` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:42.820008 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-0f29c845-6446-43a8-8713-c6e51898e8b9 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:47:42.823596 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-0f29c845-6446-43a8-8713-c6e51898e8b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:47:42.831580 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb74ac04-58f9-48ea-aa84-5127c8115c8f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 20 08:47:42.838134 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-fb74ac04-58f9-48ea-aa84-5127c8115c8f None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 20 08:47:42.838134 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:47:42.838134 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:47:42.838134 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:47:42.838134 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:47:42.838134 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:47:42.838134 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:47:42.838134 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:47:42.838134 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:47:42.838134 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:47:42.838134 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:47:42.838134 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:47:42.838134 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 20 08:47:42.838134 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 20 08:47:42.838134 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 20 08:47:42.838134 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 20 08:47:42.839031 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 20 08:47:42.839031 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 20 08:47:42.839031 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 20 08:47:42.839031 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 20 08:47:42.839031 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:47:42.839031 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:47:42.839031 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:47:42.839031 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:47:42.839031 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:47:42.839031 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:47:42.839031 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:47:42.839031 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:47:42.839031 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:47:42.839031 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:47:42.839031 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:47:42.839031 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:47:42.839793 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:47:42.839793 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:47:42.839793 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:47:42.839793 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:47:42.839793 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 20 08:47:42.839793 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 20 08:47:42.839793 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 20 08:47:42.839793 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 20 08:47:42.839793 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 20 08:47:42.839793 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 20 08:47:42.839793 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 20 08:47:42.839793 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:47:42.842462 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 49/92] 10.222.0.36 () {56 vars in 1099 bytes} [Thu Oct 20 08:47:42 2022] GET /identity/v3/roles/member => generated 84 bytes in 255 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:47:42.850646 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb74ac04-58f9-48ea-aa84-5127c8115c8f None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:42.913295 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:42.941105 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 44/93] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:47:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 20 08:47:42.960649 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb74ac04-58f9-48ea-aa84-5127c8115c8f None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:43.015739 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7db6163e-2c75-4596-8970-919789d60f87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:43.016704 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7db6163e-2c75-4596-8970-919789d60f87 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:43.017177 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7db6163e-2c75-4596-8970-919789d60f87 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:43.017642 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7db6163e-2c75-4596-8970-919789d60f87 None admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:43.018559 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7db6163e-2c75-4596-8970-919789d60f87 None admin] RBAC: Adding query filter params (name=member) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:47:43.019054 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7db6163e-2c75-4596-8970-919789d60f87 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:43.020628 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7db6163e-2c75-4596-8970-919789d60f87 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:43.034596 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 50/94] 10.222.0.36 () {56 vars in 1101 bytes} [Thu Oct 20 08:47:42 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 08:47:43.040258 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f29c845-6446-43a8-8713-c6e51898e8b9 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:43.187394 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f29c845-6446-43a8-8713-c6e51898e8b9 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:43.213112 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:43.224729 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6ee3fd4-4c75-405e-8a8f-c867d2753ad4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:43.225664 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ee3fd4-4c75-405e-8a8f-c867d2753ad4 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:43.226223 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ee3fd4-4c75-405e-8a8f-c867d2753ad4 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:43.226727 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ee3fd4-4c75-405e-8a8f-c867d2753ad4 None admin] PATH_INFO: `/v3/users/3cfc8055c0054f8f837edfcbaf0b0fe3` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:43.228072 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6ee3fd4-4c75-405e-8a8f-c867d2753ad4 None admin] RBAC: Authorizing `identity:get_user(user_id=3cfc8055c0054f8f837edfcbaf0b0fe3)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:43.229718 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6ee3fd4-4c75-405e-8a8f-c867d2753ad4 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:43.231532 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 45/95] 10.222.0.36 () {56 vars in 1177 bytes} [Thu Oct 20 08:47:43 2022] GET /identity/v3/users/3cfc8055c0054f8f837edfcbaf0b0fe3 => generated 275 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:47:43.238718 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7db6163e-2c75-4596-8970-919789d60f87 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:43.255080 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6ee3fd4-4c75-405e-8a8f-c867d2753ad4 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:43.357565 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6ee3fd4-4c75-405e-8a8f-c867d2753ad4 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:43.412022 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7db6163e-2c75-4596-8970-919789d60f87 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:43.429193 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:43.429356 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b8fd5ab-cb2b-4d0d-a087-238f7a97afca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:43.429932 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b8fd5ab-cb2b-4d0d-a087-238f7a97afca None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:43.430221 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b8fd5ab-cb2b-4d0d-a087-238f7a97afca None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:43.430518 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b8fd5ab-cb2b-4d0d-a087-238f7a97afca None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:43.451733 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b8fd5ab-cb2b-4d0d-a087-238f7a97afca None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 20 08:47:43.452581 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d74a4118-2276-4033-8806-3759d9666476 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:43.453372 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d74a4118-2276-4033-8806-3759d9666476 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:43.453794 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d74a4118-2276-4033-8806-3759d9666476 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:43.454206 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d74a4118-2276-4033-8806-3759d9666476 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:43.456866 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-1b8fd5ab-cb2b-4d0d-a087-238f7a97afca None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 20 08:47:43.456866 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:47:43.456866 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:47:43.456866 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:47:43.456866 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:47:43.456866 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:47:43.456866 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:47:43.456866 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:47:43.456866 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:47:43.456866 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:47:43.456866 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:47:43.456866 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:47:43.456866 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 20 08:47:43.456866 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 20 08:47:43.456866 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 20 08:47:43.456866 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 20 08:47:43.457905 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 20 08:47:43.457905 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 20 08:47:43.457905 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 20 08:47:43.457905 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 20 08:47:43.457905 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:47:43.457905 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:47:43.457905 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:47:43.457905 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:47:43.457905 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:47:43.457905 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:47:43.457905 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:47:43.457905 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:47:43.457905 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:47:43.457905 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:47:43.457905 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:47:43.457905 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:47:43.459452 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:47:43.459452 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:47:43.459452 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:47:43.459452 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:47:43.459452 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 20 08:47:43.459452 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 20 08:47:43.459452 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 20 08:47:43.459452 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 20 08:47:43.459452 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 20 08:47:43.459452 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 20 08:47:43.459452 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 20 08:47:43.459452 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:47:43.459452 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 46/96] 10.222.0.36 () {56 vars in 1096 bytes} [Thu Oct 20 08:47:43 2022] GET /identity/v3/roles/admin => generated 83 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:47:43.469350 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b8fd5ab-cb2b-4d0d-a087-238f7a97afca None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:43.533378 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d74a4118-2276-4033-8806-3759d9666476 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:43.535131 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d74a4118-2276-4033-8806-3759d9666476 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:43.549886 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:43.607260 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b8fd5ab-cb2b-4d0d-a087-238f7a97afca None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:43.654245 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54fbcc1a-0414-4117-8650-619280674900 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:43.654245 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54fbcc1a-0414-4117-8650-619280674900 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:43.654245 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54fbcc1a-0414-4117-8650-619280674900 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:43.654245 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54fbcc1a-0414-4117-8650-619280674900 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:43.664955 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54fbcc1a-0414-4117-8650-619280674900 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 20 08:47:43.670359 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-54fbcc1a-0414-4117-8650-619280674900 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 20 08:47:43.670359 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:47:43.670359 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:47:43.670359 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:47:43.670359 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:47:43.670359 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:47:43.670359 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:47:43.670359 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:47:43.670359 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:47:43.670359 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:47:43.670359 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:47:43.670359 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:47:43.670359 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 20 08:47:43.670359 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 20 08:47:43.670359 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 20 08:47:43.670359 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 20 08:47:43.671385 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 20 08:47:43.671385 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 20 08:47:43.671385 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 20 08:47:43.671385 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 20 08:47:43.671385 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:47:43.671385 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:47:43.671385 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:47:43.671385 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:47:43.671385 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:47:43.671385 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:47:43.671385 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:47:43.671385 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:47:43.671385 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:47:43.671385 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:47:43.671385 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:47:43.671385 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:47:43.672173 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:47:43.672173 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:47:43.672173 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:47:43.672173 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:47:43.672173 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 20 08:47:43.672173 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 20 08:47:43.672173 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 20 08:47:43.672173 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 20 08:47:43.672173 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 20 08:47:43.672173 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 20 08:47:43.672173 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 20 08:47:43.672173 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:47:43.672173 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 47/97] 10.222.0.36 () {56 vars in 1114 bytes} [Thu Oct 20 08:47:43 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:47:43.673166 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54fbcc1a-0414-4117-8650-619280674900 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:43.702427 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 51/98] 10.222.0.36 () {58 vars in 1282 bytes} [Thu Oct 20 08:47:43 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 464 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 20 08:47:43.703349 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d74a4118-2276-4033-8806-3759d9666476 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:43.727365 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:43.832674 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54fbcc1a-0414-4117-8650-619280674900 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:43.836881 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d74a4118-2276-4033-8806-3759d9666476 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:43.844719 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:43.875144 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad73abaa-6218-47f7-b596-d14cd7273fc8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:43.875894 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad73abaa-6218-47f7-b596-d14cd7273fc8 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:43.876128 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad73abaa-6218-47f7-b596-d14cd7273fc8 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:43.876401 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad73abaa-6218-47f7-b596-d14cd7273fc8 None admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:43.877064 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad73abaa-6218-47f7-b596-d14cd7273fc8 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:47:43.877372 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad73abaa-6218-47f7-b596-d14cd7273fc8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:43.878779 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad73abaa-6218-47f7-b596-d14cd7273fc8 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:43.879512 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b634ac4-e288-476e-bd07-41a8399023da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:43.880113 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b634ac4-e288-476e-bd07-41a8399023da None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:43.880950 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b634ac4-e288-476e-bd07-41a8399023da None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:43.881222 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b634ac4-e288-476e-bd07-41a8399023da None admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:43.881875 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b634ac4-e288-476e-bd07-41a8399023da None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:47:43.882175 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b634ac4-e288-476e-bd07-41a8399023da None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:43.883586 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b634ac4-e288-476e-bd07-41a8399023da None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:43.888325 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 48/99] 10.222.0.36 () {56 vars in 1099 bytes} [Thu Oct 20 08:47:43 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:47:43.889389 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad73abaa-6218-47f7-b596-d14cd7273fc8 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:43.891771 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 52/100] 10.222.0.36 () {56 vars in 1111 bytes} [Thu Oct 20 08:47:43 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:47:43.897479 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12efd090-0738-4dad-a3c2-f6a9241948b4 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:43.897768 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12efd090-0738-4dad-a3c2-f6a9241948b4 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:43.898042 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12efd090-0738-4dad-a3c2-f6a9241948b4 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:43.899468 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 53/101] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:47:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:47:43.904016 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa8e8b7-616d-422e-b064-11ea5fc46add None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:43.904306 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa8e8b7-616d-422e-b064-11ea5fc46add None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:43.904584 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa8e8b7-616d-422e-b064-11ea5fc46add None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:43.906354 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 54/102] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:47:43 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:47:43.907136 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffa8e8b7-616d-422e-b064-11ea5fc46add None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:43.997234 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:44.004187 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad73abaa-6218-47f7-b596-d14cd7273fc8 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:44.022029 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffa8e8b7-616d-422e-b064-11ea5fc46add None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:44.039639 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2572eda3-f034-4573-afa0-a6b3713a6fae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:44.040562 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2572eda3-f034-4573-afa0-a6b3713a6fae None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:44.041129 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2572eda3-f034-4573-afa0-a6b3713a6fae None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:44.041129 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2572eda3-f034-4573-afa0-a6b3713a6fae None admin] PATH_INFO: `/v3/projects/fcebd94752274a34a2068dd59f6fe700` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:44.042213 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2572eda3-f034-4573-afa0-a6b3713a6fae None admin] RBAC: Authorizing `identity:get_project(project_id=fcebd94752274a34a2068dd59f6fe700)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:44.044122 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2572eda3-f034-4573-afa0-a6b3713a6fae None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:44.050999 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 49/103] 10.222.0.36 () {56 vars in 1186 bytes} [Thu Oct 20 08:47:43 2022] GET /identity/v3/projects/fcebd94752274a34a2068dd59f6fe700 => generated 298 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:47:44.050999 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2572eda3-f034-4573-afa0-a6b3713a6fae None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:44.059970 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ccd149e-2f78-4123-babf-989bb8642a87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:44.060608 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ccd149e-2f78-4123-babf-989bb8642a87 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:44.060882 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ccd149e-2f78-4123-babf-989bb8642a87 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:44.061154 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ccd149e-2f78-4123-babf-989bb8642a87 None admin] PATH_INFO: `/v3/users/9804008bc9774f7b96bc93c72ddebd2f` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:44.062286 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ccd149e-2f78-4123-babf-989bb8642a87 None admin] RBAC: Authorizing `identity:get_user(user_id=9804008bc9774f7b96bc93c72ddebd2f)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:44.063768 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ccd149e-2f78-4123-babf-989bb8642a87 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:44.070007 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 55/104] 10.222.0.36 () {56 vars in 1177 bytes} [Thu Oct 20 08:47:43 2022] GET /identity/v3/users/9804008bc9774f7b96bc93c72ddebd2f => generated 247 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:47:44.070007 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0092a10-15b8-464c-a26c-c6a783dd2bc8 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:44.070007 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0092a10-15b8-464c-a26c-c6a783dd2bc8 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:44.070007 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0092a10-15b8-464c-a26c-c6a783dd2bc8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:44.086018 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-b0092a10-15b8-464c-a26c-c6a783dd2bc8 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:47:44.087245 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-b0092a10-15b8-464c-a26c-c6a783dd2bc8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:47:44.103074 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:44.152316 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 56/105] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:47:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 20 08:47:44.153552 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-becdb5cc-9255-42d6-9b18-d460d416626d None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:44.153833 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-becdb5cc-9255-42d6-9b18-d460d416626d None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:44.154112 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-becdb5cc-9255-42d6-9b18-d460d416626d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:44.160686 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2572eda3-f034-4573-afa0-a6b3713a6fae None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:44.171833 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-becdb5cc-9255-42d6-9b18-d460d416626d None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:47:44.173015 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-becdb5cc-9255-42d6-9b18-d460d416626d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:47:44.196320 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d17ff4-a591-452d-9475-803bc86927aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:44.196951 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d17ff4-a591-452d-9475-803bc86927aa None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:44.197238 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d17ff4-a591-452d-9475-803bc86927aa None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:44.197683 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d17ff4-a591-452d-9475-803bc86927aa None admin] PATH_INFO: `/v3/users/3cfc8055c0054f8f837edfcbaf0b0fe3` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:44.198662 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d17ff4-a591-452d-9475-803bc86927aa None admin] RBAC: Authorizing `identity:get_user(user_id=3cfc8055c0054f8f837edfcbaf0b0fe3)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:44.200107 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d17ff4-a591-452d-9475-803bc86927aa None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:44.206042 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 50/106] 10.222.0.36 () {56 vars in 1177 bytes} [Thu Oct 20 08:47:44 2022] GET /identity/v3/users/3cfc8055c0054f8f837edfcbaf0b0fe3 => generated 275 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:47:44.206042 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d17ff4-a591-452d-9475-803bc86927aa None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:44.246088 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 57/107] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:47:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 20 08:47:44.247069 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-becdb5cc-9255-42d6-9b18-d460d416626d None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:44.255633 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:44.331066 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d17ff4-a591-452d-9475-803bc86927aa None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:44.344095 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:44.361656 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-becdb5cc-9255-42d6-9b18-d460d416626d None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:44.380336 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c865216-e6ed-4545-b239-55af3e0396c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:44.382160 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c865216-e6ed-4545-b239-55af3e0396c3 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:44.382683 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c865216-e6ed-4545-b239-55af3e0396c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:44.383156 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c865216-e6ed-4545-b239-55af3e0396c3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:44.384101 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c865216-e6ed-4545-b239-55af3e0396c3 None admin] RBAC: Adding query filter params (role.id=747fbb4dfbec43baa31ec3b7761fd4b1, scope.project.id=fcebd94752274a34a2068dd59f6fe700, user.id=3cfc8055c0054f8f837edfcbaf0b0fe3) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:47:44.385162 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c865216-e6ed-4545-b239-55af3e0396c3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:44.387610 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c865216-e6ed-4545-b239-55af3e0396c3 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:44.398989 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-299d164c-7124-4c6e-8fe1-0dc6e0bb7550 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:44.399642 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 51/108] 10.222.0.36 () {56 vars in 1374 bytes} [Thu Oct 20 08:47:44 2022] GET /identity/v3/role_assignments?user.id=3cfc8055c0054f8f837edfcbaf0b0fe3&scope.project.id=fcebd94752274a34a2068dd59f6fe700&role.id=747fbb4dfbec43baa31ec3b7761fd4b1 => generated 608 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:47:44.399885 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-299d164c-7124-4c6e-8fe1-0dc6e0bb7550 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:44.400268 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-299d164c-7124-4c6e-8fe1-0dc6e0bb7550 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:44.400775 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-299d164c-7124-4c6e-8fe1-0dc6e0bb7550 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:44.401633 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dbd2f42-276e-4a2d-b347-59908397037f None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:44.402097 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dbd2f42-276e-4a2d-b347-59908397037f None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:44.402530 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dbd2f42-276e-4a2d-b347-59908397037f None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:44.404316 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 52/109] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:47:44 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:47:44.406303 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dbd2f42-276e-4a2d-b347-59908397037f None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:44.498075 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-299d164c-7124-4c6e-8fe1-0dc6e0bb7550 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:44.499989 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-299d164c-7124-4c6e-8fe1-0dc6e0bb7550 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:44.512394 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:44.539628 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dbd2f42-276e-4a2d-b347-59908397037f None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:44.579862 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7906d49c-1af8-466f-8777-42c833ea32a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:44.580706 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7906d49c-1af8-466f-8777-42c833ea32a7 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:44.580991 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7906d49c-1af8-466f-8777-42c833ea32a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:44.581267 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7906d49c-1af8-466f-8777-42c833ea32a7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:44.626386 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 58/110] 10.222.0.36 () {58 vars in 1282 bytes} [Thu Oct 20 08:47:44 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 380 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 20 08:47:44.627677 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f9db195-cc35-459b-b95e-b1dda883a6bc None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:44.627966 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f9db195-cc35-459b-b95e-b1dda883a6bc None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:44.628242 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f9db195-cc35-459b-b95e-b1dda883a6bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:44.629638 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:44.648277 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-8f9db195-cc35-459b-b95e-b1dda883a6bc None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:47:44.649467 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-8f9db195-cc35-459b-b95e-b1dda883a6bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:47:44.664104 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7906d49c-1af8-466f-8777-42c833ea32a7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:44.665574 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7906d49c-1af8-466f-8777-42c833ea32a7 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:44.719980 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 59/111] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:47:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 20 08:47:44.720821 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f9db195-cc35-459b-b95e-b1dda883a6bc None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:44.756538 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:44.776941 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 53/112] 10.222.0.36 () {58 vars in 1282 bytes} [Thu Oct 20 08:47:44 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 372 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 20 08:47:44.778127 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7906d49c-1af8-466f-8777-42c833ea32a7 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:44.826202 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f9db195-cc35-459b-b95e-b1dda883a6bc None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:44.854504 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:44.863068 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fd0d723-3184-4652-8a3d-99fac4d300f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:44.864006 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd0d723-3184-4652-8a3d-99fac4d300f0 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:44.864289 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd0d723-3184-4652-8a3d-99fac4d300f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:44.864791 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd0d723-3184-4652-8a3d-99fac4d300f0 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:44.865892 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fd0d723-3184-4652-8a3d-99fac4d300f0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:44.867398 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fd0d723-3184-4652-8a3d-99fac4d300f0 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:44.868961 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 60/113] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:47:44 2022] GET /identity/v3/domains/default => generated 199 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:47:44.874170 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fd0d723-3184-4652-8a3d-99fac4d300f0 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:44.888173 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7906d49c-1af8-466f-8777-42c833ea32a7 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:44.925621 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d05cb8d-0a92-4e27-8260-f526746d0944 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:44.926337 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d05cb8d-0a92-4e27-8260-f526746d0944 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:44.926649 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d05cb8d-0a92-4e27-8260-f526746d0944 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:44.926917 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d05cb8d-0a92-4e27-8260-f526746d0944 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:44.937028 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d05cb8d-0a92-4e27-8260-f526746d0944 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 20 08:47:44.941772 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-7d05cb8d-0a92-4e27-8260-f526746d0944 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 20 08:47:44.941772 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:47:44.941772 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:47:44.941772 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:47:44.941772 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:47:44.941772 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:47:44.941772 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:47:44.941772 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:47:44.941772 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:47:44.941772 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:47:44.941772 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:47:44.941772 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:47:44.941772 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 20 08:47:44.941772 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 20 08:47:44.941772 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 20 08:47:44.941772 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 20 08:47:44.942657 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 20 08:47:44.942657 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 20 08:47:44.942657 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 20 08:47:44.942657 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 20 08:47:44.942657 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:47:44.942657 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:47:44.942657 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:47:44.942657 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:47:44.942657 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:47:44.942657 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:47:44.942657 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:47:44.942657 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:47:44.942657 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:47:44.942657 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:47:44.942657 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:47:44.942657 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:47:44.943972 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:47:44.943972 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:47:44.943972 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:47:44.943972 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:47:44.943972 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 20 08:47:44.943972 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 20 08:47:44.943972 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 20 08:47:44.943972 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 20 08:47:44.943972 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 20 08:47:44.943972 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 20 08:47:44.943972 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 20 08:47:44.943972 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:47:44.943972 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 54/114] 10.222.0.36 () {56 vars in 1099 bytes} [Thu Oct 20 08:47:44 2022] GET /identity/v3/roles/member => generated 84 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:47:44.944620 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d05cb8d-0a92-4e27-8260-f526746d0944 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:44.983918 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fd0d723-3184-4652-8a3d-99fac4d300f0 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:44.986244 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:45.018647 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1824aa8d-9c01-4b9f-9ba0-7d0310faa861 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:45.019297 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1824aa8d-9c01-4b9f-9ba0-7d0310faa861 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:45.019542 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1824aa8d-9c01-4b9f-9ba0-7d0310faa861 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:45.019814 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1824aa8d-9c01-4b9f-9ba0-7d0310faa861 None admin] PATH_INFO: `/v3/projects/fcebd94752274a34a2068dd59f6fe700` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:45.020991 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1824aa8d-9c01-4b9f-9ba0-7d0310faa861 None admin] RBAC: Authorizing `identity:get_project(project_id=fcebd94752274a34a2068dd59f6fe700)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:45.022998 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1824aa8d-9c01-4b9f-9ba0-7d0310faa861 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:45.024619 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 61/115] 10.222.0.36 () {56 vars in 1186 bytes} [Thu Oct 20 08:47:44 2022] GET /identity/v3/projects/fcebd94752274a34a2068dd59f6fe700 => generated 298 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:47:45.025421 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1824aa8d-9c01-4b9f-9ba0-7d0310faa861 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:45.049702 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d05cb8d-0a92-4e27-8260-f526746d0944 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:45.079611 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:45.085290 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0617ed01-59a4-4dbb-8b7c-1f8ca60c5f60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:45.085912 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0617ed01-59a4-4dbb-8b7c-1f8ca60c5f60 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:45.086189 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0617ed01-59a4-4dbb-8b7c-1f8ca60c5f60 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:45.086457 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0617ed01-59a4-4dbb-8b7c-1f8ca60c5f60 None admin] PATH_INFO: `/v3/projects/fcebd94752274a34a2068dd59f6fe700` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:45.087598 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0617ed01-59a4-4dbb-8b7c-1f8ca60c5f60 None admin] RBAC: Authorizing `identity:get_project(project_id=fcebd94752274a34a2068dd59f6fe700)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:45.089044 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0617ed01-59a4-4dbb-8b7c-1f8ca60c5f60 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:45.090765 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 55/116] 10.222.0.36 () {56 vars in 1186 bytes} [Thu Oct 20 08:47:44 2022] GET /identity/v3/projects/fcebd94752274a34a2068dd59f6fe700 => generated 298 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:47:45.091534 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0617ed01-59a4-4dbb-8b7c-1f8ca60c5f60 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:45.123727 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1824aa8d-9c01-4b9f-9ba0-7d0310faa861 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:45.157408 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab35349-34c0-4633-b7df-2b5d501cada5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:45.158006 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab35349-34c0-4633-b7df-2b5d501cada5 None admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:45.158283 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab35349-34c0-4633-b7df-2b5d501cada5 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:45.158579 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab35349-34c0-4633-b7df-2b5d501cada5 None admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:45.159219 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ab35349-34c0-4633-b7df-2b5d501cada5 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:45.160666 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ab35349-34c0-4633-b7df-2b5d501cada5 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:45.188699 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0617ed01-59a4-4dbb-8b7c-1f8ca60c5f60 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:45.189111 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 62/117] 10.222.0.36 () {60 vars in 1141 bytes} [Thu Oct 20 08:47:45 2022] POST /identity/v3/projects => generated 302 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 08:47:45.189911 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab35349-34c0-4633-b7df-2b5d501cada5 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:45.226780 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:45.228651 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f75deb9-f01a-414b-8107-f29a2914c09c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:45.229282 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f75deb9-f01a-414b-8107-f29a2914c09c None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:45.229569 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f75deb9-f01a-414b-8107-f29a2914c09c None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:45.229846 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f75deb9-f01a-414b-8107-f29a2914c09c None admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:45.230564 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f75deb9-f01a-414b-8107-f29a2914c09c None admin] RBAC: Adding query filter params (name=member) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:47:45.230874 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f75deb9-f01a-414b-8107-f29a2914c09c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:45.232313 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f75deb9-f01a-414b-8107-f29a2914c09c None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:45.241915 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 56/118] 10.222.0.36 () {56 vars in 1101 bytes} [Thu Oct 20 08:47:45 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:47:45.241915 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f75deb9-f01a-414b-8107-f29a2914c09c None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:45.312685 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:45.319751 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab35349-34c0-4633-b7df-2b5d501cada5 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:45.358107 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f75deb9-f01a-414b-8107-f29a2914c09c None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:45.358648 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8421a082-b70f-4b94-8f51-3bcfc4840894 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:45.359298 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8421a082-b70f-4b94-8f51-3bcfc4840894 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:45.359737 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8421a082-b70f-4b94-8f51-3bcfc4840894 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:45.360015 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8421a082-b70f-4b94-8f51-3bcfc4840894 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:45.375497 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8421a082-b70f-4b94-8f51-3bcfc4840894 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 20 08:47:45.380589 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-8421a082-b70f-4b94-8f51-3bcfc4840894 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 20 08:47:45.380589 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:47:45.380589 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:47:45.380589 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:47:45.380589 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:47:45.380589 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:47:45.380589 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:47:45.380589 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:47:45.380589 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:47:45.380589 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:47:45.380589 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:47:45.380589 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:47:45.380589 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 20 08:47:45.380589 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 20 08:47:45.380589 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 20 08:47:45.380589 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 20 08:47:45.381464 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 20 08:47:45.381464 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 20 08:47:45.381464 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 20 08:47:45.381464 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 20 08:47:45.381464 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:47:45.381464 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:47:45.381464 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:47:45.381464 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:47:45.381464 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:47:45.381464 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:47:45.381464 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:47:45.381464 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:47:45.381464 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:47:45.381464 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:47:45.381464 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:47:45.381464 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:47:45.382214 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:47:45.382214 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:47:45.382214 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:47:45.382214 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:47:45.382214 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 20 08:47:45.382214 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 20 08:47:45.382214 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 20 08:47:45.382214 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 20 08:47:45.382214 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 20 08:47:45.382214 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 20 08:47:45.382214 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 20 08:47:45.382214 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:47:45.382214 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 63/119] 10.222.0.36 () {56 vars in 1099 bytes} [Thu Oct 20 08:47:45 2022] GET /identity/v3/roles/reader => generated 84 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:47:45.382882 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8421a082-b70f-4b94-8f51-3bcfc4840894 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:45.404117 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e24d7b75-9139-4d66-a74a-77fe7fb69f0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:45.404829 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e24d7b75-9139-4d66-a74a-77fe7fb69f0a None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:45.405116 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e24d7b75-9139-4d66-a74a-77fe7fb69f0a None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:45.405388 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e24d7b75-9139-4d66-a74a-77fe7fb69f0a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:45.406126 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e24d7b75-9139-4d66-a74a-77fe7fb69f0a None admin] RBAC: Adding query filter params (role.id=d9d3bc62f6e8453488c74856ef333bb3, scope.project.id=fcebd94752274a34a2068dd59f6fe700, user.id=9804008bc9774f7b96bc93c72ddebd2f) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:47:45.406428 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e24d7b75-9139-4d66-a74a-77fe7fb69f0a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:45.407912 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e24d7b75-9139-4d66-a74a-77fe7fb69f0a None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:45.414759 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 57/120] 10.222.0.36 () {56 vars in 1374 bytes} [Thu Oct 20 08:47:45 2022] GET /identity/v3/role_assignments?user.id=9804008bc9774f7b96bc93c72ddebd2f&scope.project.id=fcebd94752274a34a2068dd59f6fe700&role.id=d9d3bc62f6e8453488c74856ef333bb3 => generated 608 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:47:45.415344 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e24d7b75-9139-4d66-a74a-77fe7fb69f0a None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:45.489656 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:45.509813 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8421a082-b70f-4b94-8f51-3bcfc4840894 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:45.553212 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98ba0435-7c19-4db4-ae5b-d810d6ae35b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:45.553673 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e24d7b75-9139-4d66-a74a-77fe7fb69f0a None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:45.553881 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ba0435-7c19-4db4-ae5b-d810d6ae35b4 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:45.554107 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ba0435-7c19-4db4-ae5b-d810d6ae35b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:45.554429 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ba0435-7c19-4db4-ae5b-d810d6ae35b4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:45.555360 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98ba0435-7c19-4db4-ae5b-d810d6ae35b4 None admin] RBAC: Adding query filter params (role.id=b3f8d63bf4c4488288e57693c6e9804c, scope.project.id=fcebd94752274a34a2068dd59f6fe700, user.id=3cfc8055c0054f8f837edfcbaf0b0fe3) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:47:45.555677 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98ba0435-7c19-4db4-ae5b-d810d6ae35b4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:45.557267 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98ba0435-7c19-4db4-ae5b-d810d6ae35b4 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:45.564493 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 64/121] 10.222.0.36 () {56 vars in 1374 bytes} [Thu Oct 20 08:47:45 2022] GET /identity/v3/role_assignments?user.id=3cfc8055c0054f8f837edfcbaf0b0fe3&scope.project.id=fcebd94752274a34a2068dd59f6fe700&role.id=b3f8d63bf4c4488288e57693c6e9804c => generated 608 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:47:45.565511 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98ba0435-7c19-4db4-ae5b-d810d6ae35b4 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:45.587271 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:45.599319 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba20026-b46d-42ef-879e-0d3f5a5dbd47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:45.600148 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba20026-b46d-42ef-879e-0d3f5a5dbd47 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:45.600432 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba20026-b46d-42ef-879e-0d3f5a5dbd47 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:45.600707 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba20026-b46d-42ef-879e-0d3f5a5dbd47 None admin] PATH_INFO: `/v3/users/3cfc8055c0054f8f837edfcbaf0b0fe3` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:45.601945 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ba20026-b46d-42ef-879e-0d3f5a5dbd47 None admin] RBAC: Authorizing `identity:get_user(user_id=3cfc8055c0054f8f837edfcbaf0b0fe3)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:45.603765 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ba20026-b46d-42ef-879e-0d3f5a5dbd47 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:45.605865 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 58/122] 10.222.0.36 () {56 vars in 1177 bytes} [Thu Oct 20 08:47:45 2022] GET /identity/v3/users/3cfc8055c0054f8f837edfcbaf0b0fe3 => generated 275 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:47:45.611033 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba20026-b46d-42ef-879e-0d3f5a5dbd47 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:45.726157 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98ba0435-7c19-4db4-ae5b-d810d6ae35b4 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:45.765898 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba20026-b46d-42ef-879e-0d3f5a5dbd47 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:45.777704 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71432b23-b469-48d4-931f-d0f9bef90336 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:45.778750 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71432b23-b469-48d4-931f-d0f9bef90336 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:45.779243 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71432b23-b469-48d4-931f-d0f9bef90336 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:45.779713 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71432b23-b469-48d4-931f-d0f9bef90336 None admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:45.780570 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71432b23-b469-48d4-931f-d0f9bef90336 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:47:45.781063 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71432b23-b469-48d4-931f-d0f9bef90336 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:45.782744 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71432b23-b469-48d4-931f-d0f9bef90336 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:45.793411 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 65/123] 10.222.0.36 () {56 vars in 1101 bytes} [Thu Oct 20 08:47:45 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:47:45.799709 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71432b23-b469-48d4-931f-d0f9bef90336 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:45.812753 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2298c61f-fb36-4ec3-b8b2-a6ddf8011d46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:45.813739 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2298c61f-fb36-4ec3-b8b2-a6ddf8011d46 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:45.814218 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2298c61f-fb36-4ec3-b8b2-a6ddf8011d46 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:45.814797 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2298c61f-fb36-4ec3-b8b2-a6ddf8011d46 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:45.817050 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:45.897284 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:45.905481 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2298c61f-fb36-4ec3-b8b2-a6ddf8011d46 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:45.906981 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2298c61f-fb36-4ec3-b8b2-a6ddf8011d46 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:45.922696 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71432b23-b469-48d4-931f-d0f9bef90336 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:45.958197 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d858b83-c102-489c-8514-c69d18c2b234 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:45.958894 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d858b83-c102-489c-8514-c69d18c2b234 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:45.959180 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d858b83-c102-489c-8514-c69d18c2b234 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:45.959460 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d858b83-c102-489c-8514-c69d18c2b234 None admin] PATH_INFO: `/v3/users/adebc164976d449db5c64e3af6c26e54` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:45.973573 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d858b83-c102-489c-8514-c69d18c2b234 None admin] RBAC: Authorizing `identity:get_user(user_id=adebc164976d449db5c64e3af6c26e54)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:45.975130 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d858b83-c102-489c-8514-c69d18c2b234 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:45.976776 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 66/124] 10.222.0.36 () {56 vars in 1177 bytes} [Thu Oct 20 08:47:45 2022] GET /identity/v3/users/adebc164976d449db5c64e3af6c26e54 => generated 289 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 08:47:45.982167 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d858b83-c102-489c-8514-c69d18c2b234 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:46.014993 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 59/125] 10.222.0.36 () {58 vars in 1282 bytes} [Thu Oct 20 08:47:45 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 404 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 20 08:47:46.020808 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2298c61f-fb36-4ec3-b8b2-a6ddf8011d46 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:46.054070 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:46.091379 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d858b83-c102-489c-8514-c69d18c2b234 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:46.121495 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:46.128704 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2298c61f-fb36-4ec3-b8b2-a6ddf8011d46 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:46.129018 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd60583f-bf6d-4b9a-a694-efe079087d6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:46.129664 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd60583f-bf6d-4b9a-a694-efe079087d6c None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:46.129940 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd60583f-bf6d-4b9a-a694-efe079087d6c None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:46.130214 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd60583f-bf6d-4b9a-a694-efe079087d6c None admin] PATH_INFO: `/v3/projects/fcebd94752274a34a2068dd59f6fe700` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:46.131941 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd60583f-bf6d-4b9a-a694-efe079087d6c None admin] RBAC: Authorizing `identity:get_project(project_id=fcebd94752274a34a2068dd59f6fe700)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:46.133416 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd60583f-bf6d-4b9a-a694-efe079087d6c None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:46.135266 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 67/126] 10.222.0.36 () {56 vars in 1186 bytes} [Thu Oct 20 08:47:45 2022] GET /identity/v3/projects/fcebd94752274a34a2068dd59f6fe700 => generated 298 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 08:47:46.141085 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd60583f-bf6d-4b9a-a694-efe079087d6c None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:46.166925 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0edee708-79d0-46e2-9a57-e2b7d7d706a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:46.167560 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0edee708-79d0-46e2-9a57-e2b7d7d706a5 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:46.167839 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0edee708-79d0-46e2-9a57-e2b7d7d706a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:46.168108 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0edee708-79d0-46e2-9a57-e2b7d7d706a5 None admin] PATH_INFO: `/v3/projects/147a9fa19b3d4014b5dec4a105c0c0f7` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:46.169231 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0edee708-79d0-46e2-9a57-e2b7d7d706a5 None admin] RBAC: Authorizing `identity:get_project(project_id=147a9fa19b3d4014b5dec4a105c0c0f7)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:46.170699 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0edee708-79d0-46e2-9a57-e2b7d7d706a5 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:46.172544 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 60/127] 10.222.0.36 () {56 vars in 1186 bytes} [Thu Oct 20 08:47:46 2022] GET /identity/v3/projects/147a9fa19b3d4014b5dec4a105c0c0f7 => generated 312 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 08:47:46.178101 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0edee708-79d0-46e2-9a57-e2b7d7d706a5 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:46.249025 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd60583f-bf6d-4b9a-a694-efe079087d6c None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:46.277171 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0edee708-79d0-46e2-9a57-e2b7d7d706a5 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:46.284887 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f655163-5734-4dec-b3d8-1d4e1e4b99dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:46.285738 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f655163-5734-4dec-b3d8-1d4e1e4b99dd None admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:46.286339 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f655163-5734-4dec-b3d8-1d4e1e4b99dd None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:46.286669 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f655163-5734-4dec-b3d8-1d4e1e4b99dd None admin] PATH_INFO: `/v3/projects/fcebd94752274a34a2068dd59f6fe700/users/adebc164976d449db5c64e3af6c26e54/roles/9bfe29e09c61423aa168f5f0f0cbef17` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:46.288641 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f655163-5734-4dec-b3d8-1d4e1e4b99dd None admin] RBAC: Authorizing `identity:create_grant(project_id=fcebd94752274a34a2068dd59f6fe700, user_id=adebc164976d449db5c64e3af6c26e54, role_id=9bfe29e09c61423aa168f5f0f0cbef17)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:46.290212 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f655163-5734-4dec-b3d8-1d4e1e4b99dd None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:46.293293 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:46.306756 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 68/128] 10.222.0.36 () {58 vars in 1439 bytes} [Thu Oct 20 08:47:46 2022] PUT /identity/v3/projects/fcebd94752274a34a2068dd59f6fe700/users/adebc164976d449db5c64e3af6c26e54/roles/9bfe29e09c61423aa168f5f0f0cbef17 => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 08:47:46.313747 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-561d7edd-875e-4ca3-90c6-794d21c3f455 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:46.314703 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561d7edd-875e-4ca3-90c6-794d21c3f455 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:46.315162 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561d7edd-875e-4ca3-90c6-794d21c3f455 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:46.315642 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561d7edd-875e-4ca3-90c6-794d21c3f455 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:46.316601 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-561d7edd-875e-4ca3-90c6-794d21c3f455 None admin] RBAC: Adding query filter params (role.id=747fbb4dfbec43baa31ec3b7761fd4b1, scope.project.id=147a9fa19b3d4014b5dec4a105c0c0f7, user.id=3cfc8055c0054f8f837edfcbaf0b0fe3) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:47:46.317093 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-561d7edd-875e-4ca3-90c6-794d21c3f455 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:46.319448 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-561d7edd-875e-4ca3-90c6-794d21c3f455 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:46.326588 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 61/129] 10.222.0.36 () {56 vars in 1374 bytes} [Thu Oct 20 08:47:46 2022] GET /identity/v3/role_assignments?user.id=3cfc8055c0054f8f837edfcbaf0b0fe3&scope.project.id=147a9fa19b3d4014b5dec4a105c0c0f7&role.id=747fbb4dfbec43baa31ec3b7761fd4b1 => generated 608 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 08:47:47.408652 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543c3ad1-1ea2-4352-a799-5e6115a93380 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:47.409117 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543c3ad1-1ea2-4352-a799-5e6115a93380 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:47.409230 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543c3ad1-1ea2-4352-a799-5e6115a93380 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:47.410603 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 69/130] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:47:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:47:47.416410 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36af229c-d82b-4b46-a949-465b5c0d2c80 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:47.416801 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36af229c-d82b-4b46-a949-465b5c0d2c80 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:47.416962 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36af229c-d82b-4b46-a949-465b5c0d2c80 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:47.434509 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-36af229c-d82b-4b46-a949-465b5c0d2c80 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:47:47.435692 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-36af229c-d82b-4b46-a949-465b5c0d2c80 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:47:47.454795 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:47.500173 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 62/131] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:47:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 20 08:47:47.765323 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-543c3ad1-1ea2-4352-a799-5e6115a93380 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:47.863407 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-543c3ad1-1ea2-4352-a799-5e6115a93380 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:47.896781 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a588cb23-8c90-46aa-8ad8-72e9eda1caee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:47.897401 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a588cb23-8c90-46aa-8ad8-72e9eda1caee None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:47.897683 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a588cb23-8c90-46aa-8ad8-72e9eda1caee None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:47.897951 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a588cb23-8c90-46aa-8ad8-72e9eda1caee None admin] PATH_INFO: `/v3/domains/default` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:47.899055 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a588cb23-8c90-46aa-8ad8-72e9eda1caee None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:47.900504 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a588cb23-8c90-46aa-8ad8-72e9eda1caee None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:47.902050 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 70/132] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:47:47 2022] GET /identity/v3/domains/default => generated 199 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:47:47.907670 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36af229c-d82b-4b46-a949-465b5c0d2c80 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:48.002485 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36af229c-d82b-4b46-a949-465b5c0d2c80 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:48.034408 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-902a4ee4-50bd-47df-9fb6-0b80d74eeabc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:48.034973 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-902a4ee4-50bd-47df-9fb6-0b80d74eeabc None admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:48.035249 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-902a4ee4-50bd-47df-9fb6-0b80d74eeabc None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:48.035517 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-902a4ee4-50bd-47df-9fb6-0b80d74eeabc None admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:48.036164 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-902a4ee4-50bd-47df-9fb6-0b80d74eeabc None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:48.038337 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-902a4ee4-50bd-47df-9fb6-0b80d74eeabc None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:48.061405 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 63/133] 10.222.0.36 () {60 vars in 1132 bytes} [Thu Oct 20 08:47:47 2022] POST /identity/v3/users => generated 283 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 08:47:48.467133 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45bc0d92-f053-4c77-a78a-88d674fdab9f None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:48.467619 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45bc0d92-f053-4c77-a78a-88d674fdab9f None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:48.467693 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45bc0d92-f053-4c77-a78a-88d674fdab9f None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:48.469180 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 71/134] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:47:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:47:48.474916 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1ac087c-636e-4471-9129-fe1a7667a295 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:48.475382 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1ac087c-636e-4471-9129-fe1a7667a295 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:48.475458 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1ac087c-636e-4471-9129-fe1a7667a295 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:48.494137 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-d1ac087c-636e-4471-9129-fe1a7667a295 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:47:48.495514 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-d1ac087c-636e-4471-9129-fe1a7667a295 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:47:48.503871 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:48.570064 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 64/135] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:47:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 20 08:47:48.841237 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45bc0d92-f053-4c77-a78a-88d674fdab9f None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:48.907314 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:48.947983 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45bc0d92-f053-4c77-a78a-88d674fdab9f None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:48.982565 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b3d943-5d09-4ea3-a36e-94cd948e2d0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:48.983257 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b3d943-5d09-4ea3-a36e-94cd948e2d0d None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:48.983546 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b3d943-5d09-4ea3-a36e-94cd948e2d0d None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:48.983822 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b3d943-5d09-4ea3-a36e-94cd948e2d0d None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:48.998561 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92b3d943-5d09-4ea3-a36e-94cd948e2d0d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 20 08:47:49.003337 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-92b3d943-5d09-4ea3-a36e-94cd948e2d0d None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 20 08:47:49.003337 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:47:49.003337 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:47:49.003337 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:47:49.003337 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:47:49.003337 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:47:49.003337 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:47:49.003337 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:47:49.003337 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:47:49.003337 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:47:49.003337 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:47:49.003337 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:47:49.003337 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 20 08:47:49.003337 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 20 08:47:49.003337 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 20 08:47:49.003337 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 20 08:47:49.004193 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 20 08:47:49.004193 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 20 08:47:49.004193 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 20 08:47:49.004193 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 20 08:47:49.004193 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:47:49.004193 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:47:49.004193 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:47:49.004193 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:47:49.004193 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:47:49.004193 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:47:49.004193 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:47:49.004193 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:47:49.004193 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:47:49.004193 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:47:49.004193 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:47:49.004193 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:47:49.005052 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:47:49.005052 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:47:49.005052 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:47:49.005052 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:47:49.005052 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 20 08:47:49.005052 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 20 08:47:49.005052 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 20 08:47:49.005052 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 20 08:47:49.005052 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 20 08:47:49.005052 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 20 08:47:49.005052 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 20 08:47:49.005052 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:47:49.005052 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 72/136] 10.222.0.36 () {56 vars in 1099 bytes} [Thu Oct 20 08:47:48 2022] GET /identity/v3/roles/reader => generated 84 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:47:49.011119 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1ac087c-636e-4471-9129-fe1a7667a295 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:49.118665 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1ac087c-636e-4471-9129-fe1a7667a295 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:49.154160 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dcfee24-7da6-469b-b67b-517e7e9b824b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:49.154848 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dcfee24-7da6-469b-b67b-517e7e9b824b None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:49.155135 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dcfee24-7da6-469b-b67b-517e7e9b824b None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:49.155495 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dcfee24-7da6-469b-b67b-517e7e9b824b None admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:49.156202 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dcfee24-7da6-469b-b67b-517e7e9b824b None admin] RBAC: Adding query filter params (name=reader) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:47:49.156513 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dcfee24-7da6-469b-b67b-517e7e9b824b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:49.157902 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dcfee24-7da6-469b-b67b-517e7e9b824b None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:49.167230 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 65/137] 10.222.0.36 () {56 vars in 1101 bytes} [Thu Oct 20 08:47:49 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:47:49.173253 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b3d943-5d09-4ea3-a36e-94cd948e2d0d None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:49.277450 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b3d943-5d09-4ea3-a36e-94cd948e2d0d None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:49.307054 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:49.316975 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3326457-b084-4203-88cd-0b5bc5b80782 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:49.317804 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3326457-b084-4203-88cd-0b5bc5b80782 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:49.318247 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3326457-b084-4203-88cd-0b5bc5b80782 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:49.318603 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3326457-b084-4203-88cd-0b5bc5b80782 None admin] PATH_INFO: `/v3/users/adebc164976d449db5c64e3af6c26e54` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:49.319916 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3326457-b084-4203-88cd-0b5bc5b80782 None admin] RBAC: Authorizing `identity:get_user(user_id=adebc164976d449db5c64e3af6c26e54)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:49.321376 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3326457-b084-4203-88cd-0b5bc5b80782 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:49.323291 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 73/138] 10.222.0.36 () {56 vars in 1177 bytes} [Thu Oct 20 08:47:49 2022] GET /identity/v3/users/adebc164976d449db5c64e3af6c26e54 => generated 289 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:47:49.328888 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dcfee24-7da6-469b-b67b-517e7e9b824b None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:49.333247 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:49.441153 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dcfee24-7da6-469b-b67b-517e7e9b824b None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:49.477095 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad03f586-ca14-4005-ad17-34780ed15031 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:49.478462 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad03f586-ca14-4005-ad17-34780ed15031 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:49.478989 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad03f586-ca14-4005-ad17-34780ed15031 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:49.479507 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad03f586-ca14-4005-ad17-34780ed15031 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:49.550517 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad03f586-ca14-4005-ad17-34780ed15031 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:49.552522 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad03f586-ca14-4005-ad17-34780ed15031 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:49.555423 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:49.656097 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 66/139] 10.222.0.36 () {58 vars in 1282 bytes} [Thu Oct 20 08:47:49 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 328 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 20 08:47:49.662233 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3326457-b084-4203-88cd-0b5bc5b80782 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:49.766718 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3326457-b084-4203-88cd-0b5bc5b80782 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:49.801283 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6c1c905-b5d5-4311-b21d-bb36b17e2589 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:49.801909 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c1c905-b5d5-4311-b21d-bb36b17e2589 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:49.802234 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c1c905-b5d5-4311-b21d-bb36b17e2589 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:49.802728 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c1c905-b5d5-4311-b21d-bb36b17e2589 None admin] PATH_INFO: `/v3/projects/fcebd94752274a34a2068dd59f6fe700` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:49.803861 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6c1c905-b5d5-4311-b21d-bb36b17e2589 None admin] RBAC: Authorizing `identity:get_project(project_id=fcebd94752274a34a2068dd59f6fe700)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:49.805303 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6c1c905-b5d5-4311-b21d-bb36b17e2589 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:49.807120 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 74/140] 10.222.0.36 () {56 vars in 1186 bytes} [Thu Oct 20 08:47:49 2022] GET /identity/v3/projects/fcebd94752274a34a2068dd59f6fe700 => generated 298 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:47:49.813171 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad03f586-ca14-4005-ad17-34780ed15031 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:49.916416 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad03f586-ca14-4005-ad17-34780ed15031 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:49.918708 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:49.952029 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d96fc43-abfb-4c6f-b4a6-f70830752f12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:49.952636 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d96fc43-abfb-4c6f-b4a6-f70830752f12 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:49.952917 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d96fc43-abfb-4c6f-b4a6-f70830752f12 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:49.953190 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d96fc43-abfb-4c6f-b4a6-f70830752f12 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:49.953924 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d96fc43-abfb-4c6f-b4a6-f70830752f12 None admin] RBAC: Adding query filter params (role.id=9bfe29e09c61423aa168f5f0f0cbef17, scope.project.id=fcebd94752274a34a2068dd59f6fe700, user.id=adebc164976d449db5c64e3af6c26e54) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:47:49.954232 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d96fc43-abfb-4c6f-b4a6-f70830752f12 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:49.955704 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d96fc43-abfb-4c6f-b4a6-f70830752f12 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:49.962092 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 67/141] 10.222.0.36 () {56 vars in 1374 bytes} [Thu Oct 20 08:47:49 2022] GET /identity/v3/role_assignments?user.id=adebc164976d449db5c64e3af6c26e54&scope.project.id=fcebd94752274a34a2068dd59f6fe700&role.id=9bfe29e09c61423aa168f5f0f0cbef17 => generated 608 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:47:50.421776 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f196c97a-eb04-45f5-a099-0a51b42735a6 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:50.422101 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f196c97a-eb04-45f5-a099-0a51b42735a6 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:50.422332 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f196c97a-eb04-45f5-a099-0a51b42735a6 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:50.423794 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 75/142] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:47:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:47:50.429230 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aaac027-d1b3-4d28-9a33-df49c74dfd16 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:50.429524 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aaac027-d1b3-4d28-9a33-df49c74dfd16 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:50.429805 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aaac027-d1b3-4d28-9a33-df49c74dfd16 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:50.448135 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-5aaac027-d1b3-4d28-9a33-df49c74dfd16 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:47:50.449325 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-5aaac027-d1b3-4d28-9a33-df49c74dfd16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:47:50.504531 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6719b6d9-a70a-490c-80e8-9b63483cc9f7 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:50.504806 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6719b6d9-a70a-490c-80e8-9b63483cc9f7 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:50.505243 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6719b6d9-a70a-490c-80e8-9b63483cc9f7 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:50.506648 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 76/143] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:47:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 20 08:47:50.511433 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6911587c-8e90-4512-831f-a2432549fbb3 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:50.511901 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6911587c-8e90-4512-831f-a2432549fbb3 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:50.512175 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6911587c-8e90-4512-831f-a2432549fbb3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:50.524572 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 68/144] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:47:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 20 08:47:50.532044 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-6911587c-8e90-4512-831f-a2432549fbb3 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:47:50.533228 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-6911587c-8e90-4512-831f-a2432549fbb3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:47:50.580845 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc754fa9-7510-45ca-9c95-30fd37a4fc9b None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:50.581147 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc754fa9-7510-45ca-9c95-30fd37a4fc9b None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:50.581407 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc754fa9-7510-45ca-9c95-30fd37a4fc9b None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:50.583035 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 69/145] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:47:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 20 08:47:50.588429 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7a9ae0-2b4a-4693-8f19-8450a3b2b94d None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:50.588718 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7a9ae0-2b4a-4693-8f19-8450a3b2b94d None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:50.588993 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7a9ae0-2b4a-4693-8f19-8450a3b2b94d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:50.592159 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:50.606095 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 77/146] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:47:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 20 08:47:50.607867 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-9e7a9ae0-2b4a-4693-8f19-8450a3b2b94d None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:47:50.609057 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-9e7a9ae0-2b4a-4693-8f19-8450a3b2b94d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:47:50.676412 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 70/147] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:47:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 20 08:47:50.801199 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6911587c-8e90-4512-831f-a2432549fbb3 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:50.874099 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e7a9ae0-2b4a-4693-8f19-8450a3b2b94d None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:50.903332 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:50.904249 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6911587c-8e90-4512-831f-a2432549fbb3 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:50.939696 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8418057-7a9b-49b7-90d0-fba45e1317e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:50.940242 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8418057-7a9b-49b7-90d0-fba45e1317e4 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:50.940523 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8418057-7a9b-49b7-90d0-fba45e1317e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:50.940792 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8418057-7a9b-49b7-90d0-fba45e1317e4 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:50.951539 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8418057-7a9b-49b7-90d0-fba45e1317e4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 20 08:47:50.956311 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-b8418057-7a9b-49b7-90d0-fba45e1317e4 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 20 08:47:50.956311 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:47:50.956311 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:47:50.956311 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:47:50.956311 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:47:50.956311 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:47:50.956311 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:47:50.956311 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:47:50.956311 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:47:50.956311 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:47:50.956311 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:47:50.956311 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:47:50.956311 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 20 08:47:50.956311 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 20 08:47:50.956311 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 20 08:47:50.956311 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 20 08:47:50.957494 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 20 08:47:50.957494 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 20 08:47:50.957494 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 20 08:47:50.957494 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 20 08:47:50.957494 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:47:50.957494 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:47:50.957494 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:47:50.957494 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:47:50.957494 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:47:50.957494 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:47:50.957494 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:47:50.957494 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:47:50.957494 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:47:50.957494 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:47:50.957494 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:47:50.957494 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:47:50.959136 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:47:50.959136 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:47:50.959136 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:47:50.959136 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:47:50.959136 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 20 08:47:50.959136 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 20 08:47:50.959136 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 20 08:47:50.959136 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 20 08:47:50.959136 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 20 08:47:50.959136 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 20 08:47:50.959136 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 20 08:47:50.959136 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:47:50.959136 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 78/148] 10.222.0.36 () {56 vars in 1096 bytes} [Thu Oct 20 08:47:50 2022] GET /identity/v3/roles/admin => generated 83 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:47:50.959136 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8418057-7a9b-49b7-90d0-fba45e1317e4 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:50.979759 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e7a9ae0-2b4a-4693-8f19-8450a3b2b94d None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:51.008792 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:51.013119 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d11c21fb-76be-4f23-9366-965e5225507f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:51.013737 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d11c21fb-76be-4f23-9366-965e5225507f None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:51.014009 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d11c21fb-76be-4f23-9366-965e5225507f None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:51.014276 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d11c21fb-76be-4f23-9366-965e5225507f None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:51.024620 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d11c21fb-76be-4f23-9366-965e5225507f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 20 08:47:51.029377 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-d11c21fb-76be-4f23-9366-965e5225507f None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 20 08:47:51.029377 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:47:51.029377 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:47:51.029377 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:47:51.029377 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:47:51.029377 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:47:51.029377 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:47:51.029377 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:47:51.029377 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:47:51.029377 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:47:51.029377 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:47:51.029377 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:47:51.029377 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 20 08:47:51.029377 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 20 08:47:51.029377 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 20 08:47:51.029377 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 20 08:47:51.030243 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 20 08:47:51.030243 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 20 08:47:51.030243 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 20 08:47:51.030243 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 20 08:47:51.030243 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:47:51.030243 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:47:51.030243 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:47:51.030243 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:47:51.030243 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:47:51.030243 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:47:51.030243 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:47:51.030243 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:47:51.030243 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:47:51.030243 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:47:51.030243 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:47:51.030243 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:47:51.031076 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:47:51.031076 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:47:51.031076 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:47:51.031076 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:47:51.031076 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 20 08:47:51.031076 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 20 08:47:51.031076 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 20 08:47:51.031076 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 20 08:47:51.031076 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 20 08:47:51.031076 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 20 08:47:51.031076 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 20 08:47:51.031076 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:47:51.031076 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 71/149] 10.222.0.36 () {56 vars in 1114 bytes} [Thu Oct 20 08:47:50 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 20 08:47:51.031724 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d11c21fb-76be-4f23-9366-965e5225507f None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:51.059831 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8418057-7a9b-49b7-90d0-fba45e1317e4 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:51.094458 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a385dda8-62df-4061-abd8-544a3768a6ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:51.095095 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a385dda8-62df-4061-abd8-544a3768a6ee None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:51.095368 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a385dda8-62df-4061-abd8-544a3768a6ee None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:51.095637 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a385dda8-62df-4061-abd8-544a3768a6ee None admin] PATH_INFO: `/v3/domains/default` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:51.096727 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a385dda8-62df-4061-abd8-544a3768a6ee None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:51.098165 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a385dda8-62df-4061-abd8-544a3768a6ee None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:51.099775 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 79/150] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:47:50 2022] GET /identity/v3/domains/default => generated 199 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:47:51.105265 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a385dda8-62df-4061-abd8-544a3768a6ee None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:51.119965 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:51.133229 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d11c21fb-76be-4f23-9366-965e5225507f None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:51.167168 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bf7b580-db16-4fc8-a152-b6cccd41c848 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:51.167770 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf7b580-db16-4fc8-a152-b6cccd41c848 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:51.168048 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf7b580-db16-4fc8-a152-b6cccd41c848 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:51.168317 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf7b580-db16-4fc8-a152-b6cccd41c848 None admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:51.168998 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bf7b580-db16-4fc8-a152-b6cccd41c848 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:47:51.169320 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bf7b580-db16-4fc8-a152-b6cccd41c848 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:51.170756 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bf7b580-db16-4fc8-a152-b6cccd41c848 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:51.179085 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 72/151] 10.222.0.36 () {56 vars in 1099 bytes} [Thu Oct 20 08:47:51 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:47:51.179844 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bf7b580-db16-4fc8-a152-b6cccd41c848 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:51.207211 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a385dda8-62df-4061-abd8-544a3768a6ee None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:51.223752 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:51.241237 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4ca141-b0d7-4237-acef-fb02f651de7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:51.241834 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4ca141-b0d7-4237-acef-fb02f651de7b None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:51.242106 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4ca141-b0d7-4237-acef-fb02f651de7b None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:51.242374 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4ca141-b0d7-4237-acef-fb02f651de7b None admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:51.243062 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e4ca141-b0d7-4237-acef-fb02f651de7b None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:47:51.243357 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e4ca141-b0d7-4237-acef-fb02f651de7b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:51.244723 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e4ca141-b0d7-4237-acef-fb02f651de7b None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:51.252358 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 80/152] 10.222.0.36 () {56 vars in 1111 bytes} [Thu Oct 20 08:47:51 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:47:51.253138 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4ca141-b0d7-4237-acef-fb02f651de7b None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:51.280245 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bf7b580-db16-4fc8-a152-b6cccd41c848 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:51.312975 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-408e3724-6d85-4d80-a2fc-25020e61b47c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:51.313574 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-408e3724-6d85-4d80-a2fc-25020e61b47c None admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:51.313848 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-408e3724-6d85-4d80-a2fc-25020e61b47c None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:51.314119 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-408e3724-6d85-4d80-a2fc-25020e61b47c None admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:51.314796 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-408e3724-6d85-4d80-a2fc-25020e61b47c None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:51.316228 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-408e3724-6d85-4d80-a2fc-25020e61b47c None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:51.336819 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 73/153] 10.222.0.36 () {60 vars in 1132 bytes} [Thu Oct 20 08:47:51 2022] POST /identity/v3/users => generated 297 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 08:47:51.337655 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-408e3724-6d85-4d80-a2fc-25020e61b47c None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:51.347411 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:51.352982 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4ca141-b0d7-4237-acef-fb02f651de7b None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:51.397021 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf98a755-ab05-492d-879c-6f551dab9eb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:51.397677 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf98a755-ab05-492d-879c-6f551dab9eb9 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:51.397948 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf98a755-ab05-492d-879c-6f551dab9eb9 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:51.398215 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf98a755-ab05-492d-879c-6f551dab9eb9 None admin] PATH_INFO: `/v3/users/3369ee1465d54d07a96234d89bc1addf` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:51.413603 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf98a755-ab05-492d-879c-6f551dab9eb9 None admin] RBAC: Authorizing `identity:get_user(user_id=3369ee1465d54d07a96234d89bc1addf)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:51.415181 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf98a755-ab05-492d-879c-6f551dab9eb9 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:51.417089 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 81/154] 10.222.0.36 () {56 vars in 1177 bytes} [Thu Oct 20 08:47:51 2022] GET /identity/v3/users/3369ee1465d54d07a96234d89bc1addf => generated 283 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:47:51.423513 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf98a755-ab05-492d-879c-6f551dab9eb9 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:51.461061 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-408e3724-6d85-4d80-a2fc-25020e61b47c None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:51.469116 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:51.502973 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de20e866-4a53-4ce7-8073-df6a68dc7542 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:51.503613 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de20e866-4a53-4ce7-8073-df6a68dc7542 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:51.503891 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de20e866-4a53-4ce7-8073-df6a68dc7542 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:51.504158 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de20e866-4a53-4ce7-8073-df6a68dc7542 None admin] PATH_INFO: `/v3/users/3369ee1465d54d07a96234d89bc1addf` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:51.505402 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de20e866-4a53-4ce7-8073-df6a68dc7542 None admin] RBAC: Authorizing `identity:get_user(user_id=3369ee1465d54d07a96234d89bc1addf)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:51.506888 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de20e866-4a53-4ce7-8073-df6a68dc7542 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:51.508665 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 74/155] 10.222.0.36 () {56 vars in 1177 bytes} [Thu Oct 20 08:47:51 2022] GET /identity/v3/users/3369ee1465d54d07a96234d89bc1addf => generated 283 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:47:51.513989 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de20e866-4a53-4ce7-8073-df6a68dc7542 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:51.544258 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf98a755-ab05-492d-879c-6f551dab9eb9 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:51.583222 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-201bea04-a58a-43fd-8b70-66b391371261 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:51.583892 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-201bea04-a58a-43fd-8b70-66b391371261 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:51.584156 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-201bea04-a58a-43fd-8b70-66b391371261 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:51.584424 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-201bea04-a58a-43fd-8b70-66b391371261 None admin] PATH_INFO: `/v3/projects/a00517b9cca94187ac4aba4ec7468007` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:51.585919 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-201bea04-a58a-43fd-8b70-66b391371261 None admin] RBAC: Authorizing `identity:get_project(project_id=a00517b9cca94187ac4aba4ec7468007)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:51.587907 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-201bea04-a58a-43fd-8b70-66b391371261 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:51.590506 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 82/156] 10.222.0.36 () {56 vars in 1186 bytes} [Thu Oct 20 08:47:51 2022] GET /identity/v3/projects/a00517b9cca94187ac4aba4ec7468007 => generated 302 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:47:51.594676 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-201bea04-a58a-43fd-8b70-66b391371261 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:51.608262 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:51.642576 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de20e866-4a53-4ce7-8073-df6a68dc7542 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:51.690385 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112284cd-9c06-4884-95ee-14beea54e2f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:51.691050 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112284cd-9c06-4884-95ee-14beea54e2f6 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:51.691332 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112284cd-9c06-4884-95ee-14beea54e2f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:51.691604 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112284cd-9c06-4884-95ee-14beea54e2f6 None admin] PATH_INFO: `/v3/projects/a00517b9cca94187ac4aba4ec7468007` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:51.693331 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-112284cd-9c06-4884-95ee-14beea54e2f6 None admin] RBAC: Authorizing `identity:get_project(project_id=a00517b9cca94187ac4aba4ec7468007)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:51.694864 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-112284cd-9c06-4884-95ee-14beea54e2f6 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:51.696911 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 75/157] 10.222.0.36 () {56 vars in 1186 bytes} [Thu Oct 20 08:47:51 2022] GET /identity/v3/projects/a00517b9cca94187ac4aba4ec7468007 => generated 302 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 08:47:51.703172 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112284cd-9c06-4884-95ee-14beea54e2f6 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:51.725467 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-201bea04-a58a-43fd-8b70-66b391371261 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:51.736110 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:51.772350 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b83dc99-30ce-42fb-885d-8dfc29b8fb56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:51.773354 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b83dc99-30ce-42fb-885d-8dfc29b8fb56 None admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:51.773805 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b83dc99-30ce-42fb-885d-8dfc29b8fb56 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:51.774246 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b83dc99-30ce-42fb-885d-8dfc29b8fb56 None admin] PATH_INFO: `/v3/projects/a00517b9cca94187ac4aba4ec7468007/users/3369ee1465d54d07a96234d89bc1addf/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:51.777208 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b83dc99-30ce-42fb-885d-8dfc29b8fb56 None admin] RBAC: Authorizing `identity:create_grant(project_id=a00517b9cca94187ac4aba4ec7468007, user_id=3369ee1465d54d07a96234d89bc1addf, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:51.779399 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b83dc99-30ce-42fb-885d-8dfc29b8fb56 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:51.797646 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 83/158] 10.222.0.36 () {58 vars in 1439 bytes} [Thu Oct 20 08:47:51 2022] PUT /identity/v3/projects/a00517b9cca94187ac4aba4ec7468007/users/3369ee1465d54d07a96234d89bc1addf/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 203 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 08:47:51.837360 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112284cd-9c06-4884-95ee-14beea54e2f6 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:51.878087 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a69f70b2-a786-42bc-ac4f-8e2a9242b623 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:51.878805 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69f70b2-a786-42bc-ac4f-8e2a9242b623 None admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:51.879083 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69f70b2-a786-42bc-ac4f-8e2a9242b623 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:51.879354 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69f70b2-a786-42bc-ac4f-8e2a9242b623 None admin] PATH_INFO: `/v3/projects/a00517b9cca94187ac4aba4ec7468007/users/3369ee1465d54d07a96234d89bc1addf/roles/b3f8d63bf4c4488288e57693c6e9804c` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:51.882659 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a69f70b2-a786-42bc-ac4f-8e2a9242b623 None admin] RBAC: Authorizing `identity:create_grant(project_id=a00517b9cca94187ac4aba4ec7468007, user_id=3369ee1465d54d07a96234d89bc1addf, role_id=b3f8d63bf4c4488288e57693c6e9804c)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:51.884381 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a69f70b2-a786-42bc-ac4f-8e2a9242b623 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:51.901907 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 76/159] 10.222.0.36 () {58 vars in 1439 bytes} [Thu Oct 20 08:47:51 2022] PUT /identity/v3/projects/a00517b9cca94187ac4aba4ec7468007/users/3369ee1465d54d07a96234d89bc1addf/roles/b3f8d63bf4c4488288e57693c6e9804c => generated 0 bytes in 199 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 08:47:53.590414 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f774f06-9ca8-4f82-bfae-699ff2fb626a None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:53.591465 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f774f06-9ca8-4f82-bfae-699ff2fb626a None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:53.591965 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f774f06-9ca8-4f82-bfae-699ff2fb626a None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:53.593522 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 84/160] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:47:53 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:47:53.598881 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed15136-0aca-456b-ad24-a528bd3a9247 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:53.599901 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed15136-0aca-456b-ad24-a528bd3a9247 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:53.600075 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed15136-0aca-456b-ad24-a528bd3a9247 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:53.606365 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:53.621682 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-aed15136-0aca-456b-ad24-a528bd3a9247 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:47:53.623344 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-aed15136-0aca-456b-ad24-a528bd3a9247 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:47:53.701287 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 77/161] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:47:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 20 08:47:53.751680 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7245ee0-181d-4f14-a6e8-3fb096679d52 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:53.751997 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7245ee0-181d-4f14-a6e8-3fb096679d52 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:53.752211 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7245ee0-181d-4f14-a6e8-3fb096679d52 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:53.753817 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 85/162] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:47:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:47:53.759204 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51400835-ec94-44b9-9b24-ceca94328f94 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:53.759490 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51400835-ec94-44b9-9b24-ceca94328f94 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:53.759765 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51400835-ec94-44b9-9b24-ceca94328f94 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:53.778743 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-51400835-ec94-44b9-9b24-ceca94328f94 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:47:53.779923 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-51400835-ec94-44b9-9b24-ceca94328f94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:47:53.852124 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 78/163] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:47:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 20 08:47:53.974915 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7245ee0-181d-4f14-a6e8-3fb096679d52 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:53.992991 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43091752-9466-4227-bb30-f67a3e0d6508 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:53.993261 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43091752-9466-4227-bb30-f67a3e0d6508 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:53.993532 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43091752-9466-4227-bb30-f67a3e0d6508 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:53.995057 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 79/164] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:47:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 20 08:47:54.000892 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-732581c7-c933-4e76-a147-35f940f91976 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:54.001167 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-732581c7-c933-4e76-a147-35f940f91976 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:54.001437 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-732581c7-c933-4e76-a147-35f940f91976 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:54.020408 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-732581c7-c933-4e76-a147-35f940f91976 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:47:54.021647 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-732581c7-c933-4e76-a147-35f940f91976 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:47:54.094268 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7245ee0-181d-4f14-a6e8-3fb096679d52 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:54.101972 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:54.119794 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 80/165] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:47:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 20 08:47:54.136237 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-732581c7-c933-4e76-a147-35f940f91976 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:54.140717 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-304a24d0-a06b-41f5-a357-1cebd7484727 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:54.141355 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-304a24d0-a06b-41f5-a357-1cebd7484727 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:54.141631 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-304a24d0-a06b-41f5-a357-1cebd7484727 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:54.141900 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-304a24d0-a06b-41f5-a357-1cebd7484727 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:54.153025 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-304a24d0-a06b-41f5-a357-1cebd7484727 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 20 08:47:54.157872 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-304a24d0-a06b-41f5-a357-1cebd7484727 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 20 08:47:54.157872 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:47:54.157872 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:47:54.157872 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:47:54.157872 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:47:54.157872 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:47:54.157872 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:47:54.157872 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:47:54.157872 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:47:54.157872 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:47:54.157872 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:47:54.157872 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:47:54.157872 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 20 08:47:54.157872 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 20 08:47:54.157872 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 20 08:47:54.157872 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 20 08:47:54.159084 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 20 08:47:54.159084 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 20 08:47:54.159084 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 20 08:47:54.159084 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 20 08:47:54.159084 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:47:54.159084 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:47:54.159084 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:47:54.159084 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:47:54.159084 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:47:54.159084 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:47:54.159084 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:47:54.159084 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:47:54.159084 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:47:54.159084 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:47:54.159084 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:47:54.159084 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:47:54.159964 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:47:54.159964 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:47:54.159964 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:47:54.159964 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:47:54.159964 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 20 08:47:54.159964 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 20 08:47:54.159964 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 20 08:47:54.159964 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 20 08:47:54.159964 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 20 08:47:54.159964 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 20 08:47:54.159964 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 20 08:47:54.159964 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:47:54.159964 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 86/166] 10.222.0.36 () {56 vars in 1099 bytes} [Thu Oct 20 08:47:53 2022] GET /identity/v3/roles/member => generated 84 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:47:54.160757 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e227030-4a24-4bcd-865c-27bb8328cfe1 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:54.160757 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e227030-4a24-4bcd-865c-27bb8328cfe1 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:54.160757 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e227030-4a24-4bcd-865c-27bb8328cfe1 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:54.162457 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 87/167] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:47:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:47:54.165414 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e227030-4a24-4bcd-865c-27bb8328cfe1 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:54.196042 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:54.249804 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-732581c7-c933-4e76-a147-35f940f91976 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:54.278174 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e227030-4a24-4bcd-865c-27bb8328cfe1 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:54.287715 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f505582f-b80f-4cab-8338-85980d45e1ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:54.288343 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f505582f-b80f-4cab-8338-85980d45e1ba None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:54.288621 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f505582f-b80f-4cab-8338-85980d45e1ba None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:54.288886 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f505582f-b80f-4cab-8338-85980d45e1ba None admin] PATH_INFO: `/v3/domains/default` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:54.290043 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f505582f-b80f-4cab-8338-85980d45e1ba None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:54.291515 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f505582f-b80f-4cab-8338-85980d45e1ba None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:54.293080 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 81/168] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:47:54 2022] GET /identity/v3/domains/default => generated 199 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:47:54.298673 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b4579c9-9ae8-4e68-bbd6-96b545f3643c None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:54.299221 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b4579c9-9ae8-4e68-bbd6-96b545f3643c None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:54.299346 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b4579c9-9ae8-4e68-bbd6-96b545f3643c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:54.312971 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b791ef94-9474-4ecf-bd89-bd40a546048a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:54.313561 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b791ef94-9474-4ecf-bd89-bd40a546048a None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:54.313833 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b791ef94-9474-4ecf-bd89-bd40a546048a None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:54.314096 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b791ef94-9474-4ecf-bd89-bd40a546048a None admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:54.314796 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b791ef94-9474-4ecf-bd89-bd40a546048a None admin] RBAC: Adding query filter params (name=member) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:47:54.315094 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b791ef94-9474-4ecf-bd89-bd40a546048a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:54.316463 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b791ef94-9474-4ecf-bd89-bd40a546048a None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:54.316681 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-2b4579c9-9ae8-4e68-bbd6-96b545f3643c None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:47:54.318050 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-2b4579c9-9ae8-4e68-bbd6-96b545f3643c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:47:54.324747 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 88/169] 10.222.0.36 () {56 vars in 1101 bytes} [Thu Oct 20 08:47:54 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:47:54.329671 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b791ef94-9474-4ecf-bd89-bd40a546048a None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:54.381224 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:54.394333 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 82/170] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:47:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 20 08:47:54.403348 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b4579c9-9ae8-4e68-bbd6-96b545f3643c None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:54.429976 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:54.443468 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b791ef94-9474-4ecf-bd89-bd40a546048a None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:54.476390 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abcb055a-0175-4e27-963b-ccd365525eb1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:54.477023 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abcb055a-0175-4e27-963b-ccd365525eb1 None admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:54.477302 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abcb055a-0175-4e27-963b-ccd365525eb1 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:54.477579 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abcb055a-0175-4e27-963b-ccd365525eb1 None admin] PATH_INFO: `/v3/users` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:54.478232 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abcb055a-0175-4e27-963b-ccd365525eb1 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:54.479711 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abcb055a-0175-4e27-963b-ccd365525eb1 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:54.510631 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 89/171] 10.222.0.36 () {60 vars in 1132 bytes} [Thu Oct 20 08:47:54 2022] POST /identity/v3/users => generated 297 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 08:47:54.510631 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abcb055a-0175-4e27-963b-ccd365525eb1 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:54.519862 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b4579c9-9ae8-4e68-bbd6-96b545f3643c None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:54.572413 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c8bd3ee-3653-4e3f-87da-5f7d672d9cbd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:54.573427 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c8bd3ee-3653-4e3f-87da-5f7d672d9cbd None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:54.573759 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c8bd3ee-3653-4e3f-87da-5f7d672d9cbd None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:54.574182 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c8bd3ee-3653-4e3f-87da-5f7d672d9cbd None admin] PATH_INFO: `/v3/users/cacf09d14102454c817d755cd7b6ae20` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:54.594278 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c8bd3ee-3653-4e3f-87da-5f7d672d9cbd None admin] RBAC: Authorizing `identity:get_user(user_id=cacf09d14102454c817d755cd7b6ae20)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:54.596594 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c8bd3ee-3653-4e3f-87da-5f7d672d9cbd None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:54.599974 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 83/172] 10.222.0.36 () {56 vars in 1177 bytes} [Thu Oct 20 08:47:54 2022] GET /identity/v3/users/cacf09d14102454c817d755cd7b6ae20 => generated 297 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:47:54.606532 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c8bd3ee-3653-4e3f-87da-5f7d672d9cbd None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:54.626756 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abcb055a-0175-4e27-963b-ccd365525eb1 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:54.665582 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c028e8b-9189-48c7-9897-9495e3b52415 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:54.666233 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c028e8b-9189-48c7-9897-9495e3b52415 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:54.666554 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c028e8b-9189-48c7-9897-9495e3b52415 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:54.666829 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c028e8b-9189-48c7-9897-9495e3b52415 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:54.667806 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:54.679162 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c028e8b-9189-48c7-9897-9495e3b52415 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 20 08:47:54.684562 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-5c028e8b-9189-48c7-9897-9495e3b52415 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 20 08:47:54.684562 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:47:54.684562 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:47:54.684562 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:47:54.684562 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:47:54.684562 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:47:54.684562 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:47:54.684562 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:47:54.684562 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:47:54.684562 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:47:54.684562 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:47:54.684562 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:47:54.684562 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 20 08:47:54.684562 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 20 08:47:54.684562 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 20 08:47:54.684562 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 20 08:47:54.685458 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 20 08:47:54.685458 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 20 08:47:54.685458 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 20 08:47:54.685458 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 20 08:47:54.685458 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:47:54.685458 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:47:54.685458 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:47:54.685458 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:47:54.685458 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:47:54.685458 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:47:54.685458 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:47:54.685458 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:47:54.685458 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:47:54.685458 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:47:54.685458 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:47:54.685458 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:47:54.686249 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:47:54.686249 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:47:54.686249 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:47:54.686249 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:47:54.686249 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 20 08:47:54.686249 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 20 08:47:54.686249 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 20 08:47:54.686249 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 20 08:47:54.686249 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 20 08:47:54.686249 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 20 08:47:54.686249 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 20 08:47:54.686249 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:47:54.686249 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 90/173] 10.222.0.36 () {56 vars in 1096 bytes} [Thu Oct 20 08:47:54 2022] GET /identity/v3/roles/admin => generated 83 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:47:54.686943 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c028e8b-9189-48c7-9897-9495e3b52415 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:54.690027 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:54.726432 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c8bd3ee-3653-4e3f-87da-5f7d672d9cbd None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:54.767776 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba17380f-d271-4331-bc0c-71dd85804e2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:54.768331 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba17380f-d271-4331-bc0c-71dd85804e2a None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:54.768602 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba17380f-d271-4331-bc0c-71dd85804e2a None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:54.768868 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba17380f-d271-4331-bc0c-71dd85804e2a None admin] PATH_INFO: `/v3/projects/a00517b9cca94187ac4aba4ec7468007` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:54.769984 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba17380f-d271-4331-bc0c-71dd85804e2a None admin] RBAC: Authorizing `identity:get_project(project_id=a00517b9cca94187ac4aba4ec7468007)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:54.772155 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba17380f-d271-4331-bc0c-71dd85804e2a None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:54.778761 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 84/174] 10.222.0.36 () {56 vars in 1186 bytes} [Thu Oct 20 08:47:54 2022] GET /identity/v3/projects/a00517b9cca94187ac4aba4ec7468007 => generated 302 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:47:54.778761 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba17380f-d271-4331-bc0c-71dd85804e2a None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:54.800649 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c028e8b-9189-48c7-9897-9495e3b52415 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:54.843511 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db52fae8-e59a-4974-b082-4f7b0f8349a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:54.845697 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db52fae8-e59a-4974-b082-4f7b0f8349a1 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:54.845697 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db52fae8-e59a-4974-b082-4f7b0f8349a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:54.845697 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db52fae8-e59a-4974-b082-4f7b0f8349a1 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:54.857376 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db52fae8-e59a-4974-b082-4f7b0f8349a1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 20 08:47:54.862920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-db52fae8-e59a-4974-b082-4f7b0f8349a1 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 20 08:47:54.862920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:47:54.862920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:47:54.862920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:47:54.862920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:47:54.862920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:47:54.862920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:47:54.862920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:47:54.862920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:47:54.862920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:47:54.862920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:47:54.862920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:47:54.862920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 20 08:47:54.862920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 20 08:47:54.862920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 20 08:47:54.862920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 20 08:47:54.863827 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 20 08:47:54.863827 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 20 08:47:54.863827 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 20 08:47:54.863827 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 20 08:47:54.863827 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:47:54.863827 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:47:54.863827 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:47:54.863827 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:47:54.863827 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:47:54.863827 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:47:54.863827 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:47:54.863827 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:47:54.863827 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:47:54.863827 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:47:54.863827 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:47:54.863827 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:47:54.864641 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:47:54.864641 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:47:54.864641 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:47:54.864641 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:47:54.864641 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 20 08:47:54.864641 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 20 08:47:54.864641 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 20 08:47:54.864641 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 20 08:47:54.864641 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 20 08:47:54.864641 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 20 08:47:54.864641 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 20 08:47:54.864641 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:47:54.864641 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 91/175] 10.222.0.36 () {56 vars in 1114 bytes} [Thu Oct 20 08:47:54 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:47:54.865916 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db52fae8-e59a-4974-b082-4f7b0f8349a1 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:54.921881 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba17380f-d271-4331-bc0c-71dd85804e2a None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:54.937391 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:54.941668 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:54.963694 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5833fde-cf13-48be-9e88-347e91150ae6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:54.964369 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5833fde-cf13-48be-9e88-347e91150ae6 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:54.964648 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5833fde-cf13-48be-9e88-347e91150ae6 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:54.964921 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5833fde-cf13-48be-9e88-347e91150ae6 None admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:54.965645 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5833fde-cf13-48be-9e88-347e91150ae6 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:47:54.966050 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5833fde-cf13-48be-9e88-347e91150ae6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:54.967467 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5833fde-cf13-48be-9e88-347e91150ae6 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:54.978422 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 85/176] 10.222.0.36 () {56 vars in 1099 bytes} [Thu Oct 20 08:47:54 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:47:54.979319 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5833fde-cf13-48be-9e88-347e91150ae6 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:54.983865 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db52fae8-e59a-4974-b082-4f7b0f8349a1 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:55.020637 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a8ba3d6-f94e-4b38-a829-66f09afd3b86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:55.021243 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8ba3d6-f94e-4b38-a829-66f09afd3b86 None admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:55.021517 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8ba3d6-f94e-4b38-a829-66f09afd3b86 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:55.021786 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8ba3d6-f94e-4b38-a829-66f09afd3b86 None admin] PATH_INFO: `/v3/projects/a00517b9cca94187ac4aba4ec7468007/users/cacf09d14102454c817d755cd7b6ae20/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:55.023753 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a8ba3d6-f94e-4b38-a829-66f09afd3b86 None admin] RBAC: Authorizing `identity:create_grant(project_id=a00517b9cca94187ac4aba4ec7468007, user_id=cacf09d14102454c817d755cd7b6ae20, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:55.025216 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a8ba3d6-f94e-4b38-a829-66f09afd3b86 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:55.041954 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 92/177] 10.222.0.36 () {58 vars in 1439 bytes} [Thu Oct 20 08:47:54 2022] PUT /identity/v3/projects/a00517b9cca94187ac4aba4ec7468007/users/cacf09d14102454c817d755cd7b6ae20/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 08:47:55.047408 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a8ba3d6-f94e-4b38-a829-66f09afd3b86 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:55.101571 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5833fde-cf13-48be-9e88-347e91150ae6 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:55.141157 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4130e3f-9823-4ec4-827a-6c68d66d1f5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:55.141783 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4130e3f-9823-4ec4-827a-6c68d66d1f5b None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:55.142065 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4130e3f-9823-4ec4-827a-6c68d66d1f5b None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:55.142337 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4130e3f-9823-4ec4-827a-6c68d66d1f5b None admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:55.143051 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4130e3f-9823-4ec4-827a-6c68d66d1f5b None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:47:55.143415 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4130e3f-9823-4ec4-827a-6c68d66d1f5b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:55.144885 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4130e3f-9823-4ec4-827a-6c68d66d1f5b None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:55.154937 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 86/178] 10.222.0.36 () {56 vars in 1111 bytes} [Thu Oct 20 08:47:54 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:47:55.160920 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4130e3f-9823-4ec4-827a-6c68d66d1f5b None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:55.214851 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:55.280925 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a8ba3d6-f94e-4b38-a829-66f09afd3b86 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:55.297608 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4130e3f-9823-4ec4-827a-6c68d66d1f5b None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:55.314503 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:55.328515 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca884cc2-a511-4518-a4a8-8398c162545b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:55.329227 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca884cc2-a511-4518-a4a8-8398c162545b None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:55.329529 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca884cc2-a511-4518-a4a8-8398c162545b None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:55.329847 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca884cc2-a511-4518-a4a8-8398c162545b None admin] PATH_INFO: `/v3/users/3369ee1465d54d07a96234d89bc1addf` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:55.331026 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca884cc2-a511-4518-a4a8-8398c162545b None admin] RBAC: Authorizing `identity:get_user(user_id=3369ee1465d54d07a96234d89bc1addf)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:55.332885 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca884cc2-a511-4518-a4a8-8398c162545b None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:55.334855 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 93/179] 10.222.0.36 () {56 vars in 1177 bytes} [Thu Oct 20 08:47:55 2022] GET /identity/v3/users/3369ee1465d54d07a96234d89bc1addf => generated 283 bytes in 292 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:47:55.341419 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca884cc2-a511-4518-a4a8-8398c162545b None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:55.361225 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a52d5979-a56b-4f6e-945f-7c5ac27628c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:55.361856 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52d5979-a56b-4f6e-945f-7c5ac27628c0 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:55.362137 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52d5979-a56b-4f6e-945f-7c5ac27628c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:55.362411 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52d5979-a56b-4f6e-945f-7c5ac27628c0 None admin] PATH_INFO: `/v3/users/3369ee1465d54d07a96234d89bc1addf` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:55.369410 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a52d5979-a56b-4f6e-945f-7c5ac27628c0 None admin] RBAC: Authorizing `identity:get_user(user_id=3369ee1465d54d07a96234d89bc1addf)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:55.370912 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a52d5979-a56b-4f6e-945f-7c5ac27628c0 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:55.372536 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 87/180] 10.222.0.36 () {56 vars in 1177 bytes} [Thu Oct 20 08:47:55 2022] GET /identity/v3/users/3369ee1465d54d07a96234d89bc1addf => generated 283 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:47:55.378617 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a52d5979-a56b-4f6e-945f-7c5ac27628c0 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:55.472534 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca884cc2-a511-4518-a4a8-8398c162545b None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:55.494902 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a52d5979-a56b-4f6e-945f-7c5ac27628c0 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:55.504686 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:55.520704 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64354499-8f24-4bdd-b5b9-8d0d34f2252a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:55.522145 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64354499-8f24-4bdd-b5b9-8d0d34f2252a None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:55.522887 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64354499-8f24-4bdd-b5b9-8d0d34f2252a None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:55.523356 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64354499-8f24-4bdd-b5b9-8d0d34f2252a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:55.543651 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-063df214-b3de-4207-b192-99c36999c001 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:55.544236 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-063df214-b3de-4207-b192-99c36999c001 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:55.544525 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-063df214-b3de-4207-b192-99c36999c001 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:55.544798 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-063df214-b3de-4207-b192-99c36999c001 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:55.653766 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:55.667565 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64354499-8f24-4bdd-b5b9-8d0d34f2252a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:55.669072 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64354499-8f24-4bdd-b5b9-8d0d34f2252a None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:55.679223 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-063df214-b3de-4207-b192-99c36999c001 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:55.681254 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-063df214-b3de-4207-b192-99c36999c001 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:55.788873 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 94/181] 10.222.0.36 () {58 vars in 1282 bytes} [Thu Oct 20 08:47:55 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 448 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 20 08:47:55.794290 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64354499-8f24-4bdd-b5b9-8d0d34f2252a None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:55.902402 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:55.914280 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64354499-8f24-4bdd-b5b9-8d0d34f2252a None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:55.931144 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 88/182] 10.222.0.36 () {58 vars in 1282 bytes} [Thu Oct 20 08:47:55 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 553 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 20 08:47:55.936568 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-063df214-b3de-4207-b192-99c36999c001 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:55.947720 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:55.967741 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4c61b71-3223-4926-8da1-e296cd268310 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:55.968802 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c61b71-3223-4926-8da1-e296cd268310 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:55.969209 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c61b71-3223-4926-8da1-e296cd268310 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:55.969629 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c61b71-3223-4926-8da1-e296cd268310 None admin] PATH_INFO: `/v3/projects/a00517b9cca94187ac4aba4ec7468007` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:55.971030 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4c61b71-3223-4926-8da1-e296cd268310 None admin] RBAC: Authorizing `identity:get_project(project_id=a00517b9cca94187ac4aba4ec7468007)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:55.973139 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4c61b71-3223-4926-8da1-e296cd268310 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:55.976660 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 95/183] 10.222.0.36 () {56 vars in 1186 bytes} [Thu Oct 20 08:47:55 2022] GET /identity/v3/projects/a00517b9cca94187ac4aba4ec7468007 => generated 302 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 08:47:55.982778 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4c61b71-3223-4926-8da1-e296cd268310 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:56.072040 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-063df214-b3de-4207-b192-99c36999c001 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:56.126745 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59118481-56e1-4a36-85de-dc7e1195d84c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:56.127377 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59118481-56e1-4a36-85de-dc7e1195d84c None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:56.127649 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59118481-56e1-4a36-85de-dc7e1195d84c None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:56.127913 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59118481-56e1-4a36-85de-dc7e1195d84c None admin] PATH_INFO: `/v3/projects/a00517b9cca94187ac4aba4ec7468007` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:56.131902 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59118481-56e1-4a36-85de-dc7e1195d84c None admin] RBAC: Authorizing `identity:get_project(project_id=a00517b9cca94187ac4aba4ec7468007)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:56.131902 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59118481-56e1-4a36-85de-dc7e1195d84c None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:56.132697 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 89/184] 10.222.0.36 () {56 vars in 1186 bytes} [Thu Oct 20 08:47:55 2022] GET /identity/v3/projects/a00517b9cca94187ac4aba4ec7468007 => generated 302 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 08:47:56.139340 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59118481-56e1-4a36-85de-dc7e1195d84c None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:56.160945 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4c61b71-3223-4926-8da1-e296cd268310 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:56.229870 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29e2a095-3c8e-4cd2-a787-157ad80988d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:56.230499 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e2a095-3c8e-4cd2-a787-157ad80988d6 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:56.230779 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e2a095-3c8e-4cd2-a787-157ad80988d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:56.231070 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e2a095-3c8e-4cd2-a787-157ad80988d6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:56.231763 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29e2a095-3c8e-4cd2-a787-157ad80988d6 None admin] RBAC: Adding query filter params (role.id=d9d3bc62f6e8453488c74856ef333bb3, scope.project.id=a00517b9cca94187ac4aba4ec7468007, user.id=3369ee1465d54d07a96234d89bc1addf) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:47:56.232063 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29e2a095-3c8e-4cd2-a787-157ad80988d6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:56.233501 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29e2a095-3c8e-4cd2-a787-157ad80988d6 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:56.240963 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 96/185] 10.222.0.36 () {56 vars in 1374 bytes} [Thu Oct 20 08:47:55 2022] GET /identity/v3/role_assignments?user.id=3369ee1465d54d07a96234d89bc1addf&scope.project.id=a00517b9cca94187ac4aba4ec7468007&role.id=d9d3bc62f6e8453488c74856ef333bb3 => generated 608 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 08:47:56.280288 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:56.305401 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59118481-56e1-4a36-85de-dc7e1195d84c None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:56.348072 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a424af2-77e8-4349-a692-eaa6e9c27112 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:56.348616 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a424af2-77e8-4349-a692-eaa6e9c27112 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:56.348885 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a424af2-77e8-4349-a692-eaa6e9c27112 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:56.349703 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a424af2-77e8-4349-a692-eaa6e9c27112 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:56.350449 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a424af2-77e8-4349-a692-eaa6e9c27112 None admin] RBAC: Adding query filter params (role.id=b3f8d63bf4c4488288e57693c6e9804c, scope.project.id=a00517b9cca94187ac4aba4ec7468007, user.id=3369ee1465d54d07a96234d89bc1addf) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:47:56.350780 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a424af2-77e8-4349-a692-eaa6e9c27112 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:56.352312 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a424af2-77e8-4349-a692-eaa6e9c27112 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:56.362070 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 90/186] 10.222.0.36 () {56 vars in 1374 bytes} [Thu Oct 20 08:47:56 2022] GET /identity/v3/role_assignments?user.id=3369ee1465d54d07a96234d89bc1addf&scope.project.id=a00517b9cca94187ac4aba4ec7468007&role.id=b3f8d63bf4c4488288e57693c6e9804c => generated 608 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:47:57.129597 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e93fc10-d763-45df-b2d4-599622526fbe None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:57.130156 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e93fc10-d763-45df-b2d4-599622526fbe None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:57.130156 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e93fc10-d763-45df-b2d4-599622526fbe None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:57.131222 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c01a2329-ade3-4eef-8063-95b5117c434f None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:57.131627 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c01a2329-ade3-4eef-8063-95b5117c434f None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:57.131832 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c01a2329-ade3-4eef-8063-95b5117c434f None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:57.132239 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 97/187] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:47:57 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:47:57.133625 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 91/188] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:47:57 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:47:57.137112 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d780e08-34a9-4f80-95e3-0c9e90cad219 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:57.137549 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d780e08-34a9-4f80-95e3-0c9e90cad219 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:57.137663 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d780e08-34a9-4f80-95e3-0c9e90cad219 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:57.139098 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-538ea5ab-61aa-469a-97ff-f08ad8be95f9 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:57.139384 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-538ea5ab-61aa-469a-97ff-f08ad8be95f9 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:57.139658 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-538ea5ab-61aa-469a-97ff-f08ad8be95f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:57.159079 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-538ea5ab-61aa-469a-97ff-f08ad8be95f9 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:47:57.159283 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-6d780e08-34a9-4f80-95e3-0c9e90cad219 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:47:57.160268 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-538ea5ab-61aa-469a-97ff-f08ad8be95f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:47:57.160371 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-6d780e08-34a9-4f80-95e3-0c9e90cad219 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:47:57.182677 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:57.228511 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 92/189] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:47:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 20 08:47:57.243067 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 98/190] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:47:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 20 08:47:57.497123 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-538ea5ab-61aa-469a-97ff-f08ad8be95f9 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:57.514191 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a1243d-4e24-4668-9381-61e271fd2c4f None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:57.514497 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a1243d-4e24-4668-9381-61e271fd2c4f None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:57.514766 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a1243d-4e24-4668-9381-61e271fd2c4f None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:57.516281 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 99/191] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:47:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:47:57.518229 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4a1243d-4e24-4668-9381-61e271fd2c4f None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:57.564727 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:57.594283 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-538ea5ab-61aa-469a-97ff-f08ad8be95f9 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:57.619089 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4a1243d-4e24-4668-9381-61e271fd2c4f None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:57.628364 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edf3249b-7c70-4dd8-92de-66a82b240c76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:57.629328 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edf3249b-7c70-4dd8-92de-66a82b240c76 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:57.629602 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edf3249b-7c70-4dd8-92de-66a82b240c76 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:57.629870 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edf3249b-7c70-4dd8-92de-66a82b240c76 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:57.631006 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edf3249b-7c70-4dd8-92de-66a82b240c76 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:57.632452 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edf3249b-7c70-4dd8-92de-66a82b240c76 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:57.634170 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 93/192] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:47:57 2022] GET /identity/v3/domains/default => generated 199 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:47:57.635105 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243d3acb-8107-4a9b-b69d-92fd21afe767 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:57.635381 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243d3acb-8107-4a9b-b69d-92fd21afe767 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:57.635648 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243d3acb-8107-4a9b-b69d-92fd21afe767 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:57.653107 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-243d3acb-8107-4a9b-b69d-92fd21afe767 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:47:57.654254 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-243d3acb-8107-4a9b-b69d-92fd21afe767 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:47:57.656593 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5120dd02-311b-4351-bebb-6fdc69f984ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:57.657217 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5120dd02-311b-4351-bebb-6fdc69f984ee None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:57.657492 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5120dd02-311b-4351-bebb-6fdc69f984ee None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:57.657761 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5120dd02-311b-4351-bebb-6fdc69f984ee None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:57.668184 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5120dd02-311b-4351-bebb-6fdc69f984ee None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 20 08:47:57.672968 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-5120dd02-311b-4351-bebb-6fdc69f984ee None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 20 08:47:57.672968 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:47:57.672968 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:47:57.672968 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:47:57.672968 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:47:57.672968 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:47:57.672968 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:47:57.672968 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:47:57.672968 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:47:57.672968 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:47:57.672968 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:47:57.672968 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:47:57.672968 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 20 08:47:57.672968 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 20 08:47:57.672968 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 20 08:47:57.672968 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 20 08:47:57.673922 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 20 08:47:57.673922 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 20 08:47:57.673922 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 20 08:47:57.673922 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 20 08:47:57.673922 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:47:57.673922 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:47:57.673922 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:47:57.673922 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:47:57.673922 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:47:57.673922 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:47:57.673922 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:47:57.673922 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:47:57.673922 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:47:57.673922 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:47:57.673922 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:47:57.673922 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:47:57.679417 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:47:57.679417 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:47:57.679417 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:47:57.679417 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:47:57.679417 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 20 08:47:57.679417 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 20 08:47:57.679417 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 20 08:47:57.679417 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 20 08:47:57.679417 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 20 08:47:57.679417 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 20 08:47:57.679417 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 20 08:47:57.679417 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:47:57.679417 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 100/193] 10.222.0.36 () {56 vars in 1099 bytes} [Thu Oct 20 08:47:57 2022] GET /identity/v3/roles/reader => generated 84 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 20 08:47:57.679417 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5120dd02-311b-4351-bebb-6fdc69f984ee None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:57.690588 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:57.721465 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 94/194] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:47:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 20 08:47:57.722285 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-243d3acb-8107-4a9b-b69d-92fd21afe767 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:57.777580 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5120dd02-311b-4351-bebb-6fdc69f984ee None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:57.788973 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:57.818428 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ccdc6c-7a0e-4150-a510-52534ad492cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:57.819331 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ccdc6c-7a0e-4150-a510-52534ad492cc None admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:57.819615 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ccdc6c-7a0e-4150-a510-52534ad492cc None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:57.820132 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ccdc6c-7a0e-4150-a510-52534ad492cc None admin] PATH_INFO: `/v3/users` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:57.821191 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93ccdc6c-7a0e-4150-a510-52534ad492cc None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:57.822994 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93ccdc6c-7a0e-4150-a510-52534ad492cc None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:57.823567 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-243d3acb-8107-4a9b-b69d-92fd21afe767 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:57.848306 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 101/195] 10.222.0.36 () {60 vars in 1132 bytes} [Thu Oct 20 08:47:57 2022] POST /identity/v3/users => generated 293 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 08:47:57.861635 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f0ea01-68e1-42da-889c-4207a9035daf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:57.862267 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f0ea01-68e1-42da-889c-4207a9035daf None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:57.862586 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f0ea01-68e1-42da-889c-4207a9035daf None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:57.862904 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f0ea01-68e1-42da-889c-4207a9035daf None admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:57.863598 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47f0ea01-68e1-42da-889c-4207a9035daf None admin] RBAC: Adding query filter params (name=reader) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:47:57.863907 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47f0ea01-68e1-42da-889c-4207a9035daf None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:57.865939 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47f0ea01-68e1-42da-889c-4207a9035daf None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:57.881020 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 95/196] 10.222.0.36 () {56 vars in 1101 bytes} [Thu Oct 20 08:47:57 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:47:57.883644 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ccdc6c-7a0e-4150-a510-52534ad492cc None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:57.962270 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:57.986972 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f0ea01-68e1-42da-889c-4207a9035daf None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:57.998885 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ccdc6c-7a0e-4150-a510-52534ad492cc None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:58.040364 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8664442-fd30-4961-9d3a-3295dfe6b0c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:58.040927 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8664442-fd30-4961-9d3a-3295dfe6b0c1 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:58.041206 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8664442-fd30-4961-9d3a-3295dfe6b0c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:58.041478 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8664442-fd30-4961-9d3a-3295dfe6b0c1 None admin] PATH_INFO: `/v3/users/cae79f71b1364dca9beb3590abc86b3f` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:58.057135 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8664442-fd30-4961-9d3a-3295dfe6b0c1 None admin] RBAC: Authorizing `identity:get_user(user_id=cae79f71b1364dca9beb3590abc86b3f)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:58.058686 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8664442-fd30-4961-9d3a-3295dfe6b0c1 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:58.060728 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 102/197] 10.222.0.36 () {56 vars in 1177 bytes} [Thu Oct 20 08:47:57 2022] GET /identity/v3/users/cae79f71b1364dca9beb3590abc86b3f => generated 297 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 08:47:58.065863 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8664442-fd30-4961-9d3a-3295dfe6b0c1 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:58.107323 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f0ea01-68e1-42da-889c-4207a9035daf None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:58.144583 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:58.151917 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72adbb53-9ddc-458b-adb5-e6d6fd7cc363 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:58.153433 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72adbb53-9ddc-458b-adb5-e6d6fd7cc363 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:58.153991 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72adbb53-9ddc-458b-adb5-e6d6fd7cc363 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:58.154274 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72adbb53-9ddc-458b-adb5-e6d6fd7cc363 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:58.170259 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72adbb53-9ddc-458b-adb5-e6d6fd7cc363 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 20 08:47:58.176219 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-72adbb53-9ddc-458b-adb5-e6d6fd7cc363 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 20 08:47:58.176219 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:47:58.176219 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:47:58.176219 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:47:58.176219 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:47:58.176219 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:47:58.176219 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:47:58.176219 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:47:58.176219 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:47:58.176219 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:47:58.176219 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:47:58.176219 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:47:58.176219 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 20 08:47:58.176219 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 20 08:47:58.176219 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 20 08:47:58.176219 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 20 08:47:58.177111 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 20 08:47:58.177111 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 20 08:47:58.177111 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 20 08:47:58.177111 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 20 08:47:58.177111 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:47:58.177111 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:47:58.177111 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:47:58.177111 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:47:58.177111 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:47:58.177111 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:47:58.177111 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:47:58.177111 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:47:58.177111 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:47:58.177111 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:47:58.177111 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:47:58.177111 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:47:58.178457 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:47:58.178457 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:47:58.178457 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:47:58.178457 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:47:58.178457 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 20 08:47:58.178457 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 20 08:47:58.178457 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 20 08:47:58.178457 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 20 08:47:58.178457 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 20 08:47:58.178457 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 20 08:47:58.178457 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 20 08:47:58.178457 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:47:58.178457 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 96/198] 10.222.0.36 () {56 vars in 1099 bytes} [Thu Oct 20 08:47:57 2022] GET /identity/v3/roles/member => generated 84 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 20 08:47:58.184120 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72adbb53-9ddc-458b-adb5-e6d6fd7cc363 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:58.195017 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8664442-fd30-4961-9d3a-3295dfe6b0c1 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:58.226528 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:58.238901 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a07ca4e-4956-4039-9938-d4acd77f05cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:58.239545 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a07ca4e-4956-4039-9938-d4acd77f05cb None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:58.240420 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a07ca4e-4956-4039-9938-d4acd77f05cb None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:58.240698 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a07ca4e-4956-4039-9938-d4acd77f05cb None admin] PATH_INFO: `/v3/projects/a00517b9cca94187ac4aba4ec7468007` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:58.241870 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a07ca4e-4956-4039-9938-d4acd77f05cb None admin] RBAC: Authorizing `identity:get_project(project_id=a00517b9cca94187ac4aba4ec7468007)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:58.244063 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a07ca4e-4956-4039-9938-d4acd77f05cb None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:58.245174 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 103/199] 10.222.0.36 () {56 vars in 1186 bytes} [Thu Oct 20 08:47:58 2022] GET /identity/v3/projects/a00517b9cca94187ac4aba4ec7468007 => generated 302 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 08:47:58.252006 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a07ca4e-4956-4039-9938-d4acd77f05cb None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:58.307452 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72adbb53-9ddc-458b-adb5-e6d6fd7cc363 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:58.347447 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1b3fc43-875b-49fa-977d-bc138d3ab170 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:58.348046 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b3fc43-875b-49fa-977d-bc138d3ab170 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:58.348319 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b3fc43-875b-49fa-977d-bc138d3ab170 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:58.348580 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b3fc43-875b-49fa-977d-bc138d3ab170 None admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:58.349758 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1b3fc43-875b-49fa-977d-bc138d3ab170 None admin] RBAC: Adding query filter params (name=member) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:47:58.350056 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1b3fc43-875b-49fa-977d-bc138d3ab170 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:58.351482 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1b3fc43-875b-49fa-977d-bc138d3ab170 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:58.363092 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 97/200] 10.222.0.36 () {56 vars in 1101 bytes} [Thu Oct 20 08:47:58 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 08:47:58.371464 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1b3fc43-875b-49fa-977d-bc138d3ab170 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:58.400421 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a07ca4e-4956-4039-9938-d4acd77f05cb None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:58.440796 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:58.449134 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7993e4be-6c32-4b31-8c55-aae86cc3c330 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:58.450336 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7993e4be-6c32-4b31-8c55-aae86cc3c330 None admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:58.451077 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7993e4be-6c32-4b31-8c55-aae86cc3c330 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:58.451936 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7993e4be-6c32-4b31-8c55-aae86cc3c330 None admin] PATH_INFO: `/v3/projects/a00517b9cca94187ac4aba4ec7468007/users/cae79f71b1364dca9beb3590abc86b3f/roles/9bfe29e09c61423aa168f5f0f0cbef17` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:58.454348 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7993e4be-6c32-4b31-8c55-aae86cc3c330 None admin] RBAC: Authorizing `identity:create_grant(project_id=a00517b9cca94187ac4aba4ec7468007, user_id=cae79f71b1364dca9beb3590abc86b3f, role_id=9bfe29e09c61423aa168f5f0f0cbef17)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:58.457248 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7993e4be-6c32-4b31-8c55-aae86cc3c330 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:58.480325 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 104/201] 10.222.0.36 () {58 vars in 1439 bytes} [Thu Oct 20 08:47:58 2022] PUT /identity/v3/projects/a00517b9cca94187ac4aba4ec7468007/users/cae79f71b1364dca9beb3590abc86b3f/roles/9bfe29e09c61423aa168f5f0f0cbef17 => generated 0 bytes in 229 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 08:47:58.496368 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1b3fc43-875b-49fa-977d-bc138d3ab170 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:58.534832 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeb9f506-782b-4d1b-b15a-799c7e89727c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:58.535487 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb9f506-782b-4d1b-b15a-799c7e89727c None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:58.535768 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb9f506-782b-4d1b-b15a-799c7e89727c None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:58.536038 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb9f506-782b-4d1b-b15a-799c7e89727c None admin] PATH_INFO: `/v3/users/cacf09d14102454c817d755cd7b6ae20` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:58.537852 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeb9f506-782b-4d1b-b15a-799c7e89727c None admin] RBAC: Authorizing `identity:get_user(user_id=cacf09d14102454c817d755cd7b6ae20)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:58.539460 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeb9f506-782b-4d1b-b15a-799c7e89727c None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:58.541167 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 98/202] 10.222.0.36 () {56 vars in 1177 bytes} [Thu Oct 20 08:47:58 2022] GET /identity/v3/users/cacf09d14102454c817d755cd7b6ae20 => generated 297 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:47:58.547219 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7993e4be-6c32-4b31-8c55-aae86cc3c330 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:58.611341 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:58.663864 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7993e4be-6c32-4b31-8c55-aae86cc3c330 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:58.704510 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab86079c-fdc7-4c83-a0dc-c591b5392693 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:58.705123 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab86079c-fdc7-4c83-a0dc-c591b5392693 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:58.705402 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab86079c-fdc7-4c83-a0dc-c591b5392693 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:58.705678 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab86079c-fdc7-4c83-a0dc-c591b5392693 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:58.782906 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab86079c-fdc7-4c83-a0dc-c591b5392693 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:58.784398 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab86079c-fdc7-4c83-a0dc-c591b5392693 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:58.845921 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:58.894831 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 105/203] 10.222.0.36 () {58 vars in 1282 bytes} [Thu Oct 20 08:47:58 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 348 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 20 08:47:58.900720 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeb9f506-782b-4d1b-b15a-799c7e89727c None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:59.010812 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeb9f506-782b-4d1b-b15a-799c7e89727c None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:59.030380 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:47:59.046949 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5888519-e3d2-4536-a4e4-306e320ab795 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:59.047571 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5888519-e3d2-4536-a4e4-306e320ab795 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:59.047846 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5888519-e3d2-4536-a4e4-306e320ab795 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:59.048109 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5888519-e3d2-4536-a4e4-306e320ab795 None admin] PATH_INFO: `/v3/projects/a00517b9cca94187ac4aba4ec7468007` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:59.049221 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5888519-e3d2-4536-a4e4-306e320ab795 None admin] RBAC: Authorizing `identity:get_project(project_id=a00517b9cca94187ac4aba4ec7468007)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:59.050688 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5888519-e3d2-4536-a4e4-306e320ab795 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:59.052841 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 99/204] 10.222.0.36 () {56 vars in 1186 bytes} [Thu Oct 20 08:47:58 2022] GET /identity/v3/projects/a00517b9cca94187ac4aba4ec7468007 => generated 302 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:47:59.058715 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab86079c-fdc7-4c83-a0dc-c591b5392693 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:47:59.161469 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab86079c-fdc7-4c83-a0dc-c591b5392693 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:47:59.197297 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa8b08f7-d4ea-4378-8fb1-fb2bd5504449 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:47:59.198013 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8b08f7-d4ea-4378-8fb1-fb2bd5504449 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:47:59.198185 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8b08f7-d4ea-4378-8fb1-fb2bd5504449 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:47:59.198435 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8b08f7-d4ea-4378-8fb1-fb2bd5504449 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:47:59.199350 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa8b08f7-d4ea-4378-8fb1-fb2bd5504449 None admin] RBAC: Adding query filter params (role.id=747fbb4dfbec43baa31ec3b7761fd4b1, scope.project.id=a00517b9cca94187ac4aba4ec7468007, user.id=cacf09d14102454c817d755cd7b6ae20) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:47:59.199670 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa8b08f7-d4ea-4378-8fb1-fb2bd5504449 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:47:59.201320 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa8b08f7-d4ea-4378-8fb1-fb2bd5504449 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:47:59.207801 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 106/205] 10.222.0.36 () {56 vars in 1374 bytes} [Thu Oct 20 08:47:59 2022] GET /identity/v3/role_assignments?user.id=cacf09d14102454c817d755cd7b6ae20&scope.project.id=a00517b9cca94187ac4aba4ec7468007&role.id=747fbb4dfbec43baa31ec3b7761fd4b1 => generated 608 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:00.174533 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7ca083-1295-4b73-96cf-63a8057504c0 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:00.174533 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7ca083-1295-4b73-96cf-63a8057504c0 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:00.174533 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7ca083-1295-4b73-96cf-63a8057504c0 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:00.175636 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 100/206] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:48:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:48:00.181293 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f34ef6-6267-4115-b37d-09163f07721d None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:00.181573 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f34ef6-6267-4115-b37d-09163f07721d None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:00.182007 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f34ef6-6267-4115-b37d-09163f07721d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:00.200232 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-f6f34ef6-6267-4115-b37d-09163f07721d None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:48:00.202185 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-f6f34ef6-6267-4115-b37d-09163f07721d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:48:00.217747 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:00.268259 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747b77ad-764d-46f0-90cd-8f8eee49fd8a None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:00.268662 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747b77ad-764d-46f0-90cd-8f8eee49fd8a None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:00.269081 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747b77ad-764d-46f0-90cd-8f8eee49fd8a None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:00.270809 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 101/207] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:48:00 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 20 08:48:00.275837 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 107/208] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:48:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 20 08:48:00.276392 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3069be5d-488d-47ad-bc35-1da8624a3d75 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:00.276687 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3069be5d-488d-47ad-bc35-1da8624a3d75 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:00.276942 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3069be5d-488d-47ad-bc35-1da8624a3d75 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:00.298365 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-3069be5d-488d-47ad-bc35-1da8624a3d75 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:48:00.299585 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-3069be5d-488d-47ad-bc35-1da8624a3d75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:48:00.369982 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 102/209] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:48:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 20 08:48:00.538078 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f34ef6-6267-4115-b37d-09163f07721d None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:00.638955 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3069be5d-488d-47ad-bc35-1da8624a3d75 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:00.645584 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f34ef6-6267-4115-b37d-09163f07721d None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:00.681458 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0421aab8-6307-4080-9599-996549502948 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:00.682081 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0421aab8-6307-4080-9599-996549502948 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:00.682351 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0421aab8-6307-4080-9599-996549502948 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:00.682833 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0421aab8-6307-4080-9599-996549502948 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:00.693610 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0421aab8-6307-4080-9599-996549502948 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 20 08:48:00.698353 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-0421aab8-6307-4080-9599-996549502948 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 20 08:48:00.698353 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:00.698353 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:00.698353 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:00.698353 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:00.698353 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:00.698353 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:00.698353 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:00.698353 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:00.698353 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:00.698353 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:00.698353 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:00.698353 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 20 08:48:00.698353 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 20 08:48:00.698353 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 20 08:48:00.698353 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 20 08:48:00.699268 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 20 08:48:00.699268 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 20 08:48:00.699268 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 20 08:48:00.699268 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 20 08:48:00.699268 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:00.699268 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:00.699268 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:00.699268 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:00.699268 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:00.699268 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:00.699268 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:00.699268 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:00.699268 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:00.699268 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:00.699268 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:00.699268 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:00.700163 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:00.700163 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:00.700163 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:00.700163 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:00.700163 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 20 08:48:00.700163 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 20 08:48:00.700163 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 20 08:48:00.700163 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 20 08:48:00.700163 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 20 08:48:00.700163 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 20 08:48:00.700163 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 20 08:48:00.700163 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:48:00.700163 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 108/210] 10.222.0.36 () {56 vars in 1099 bytes} [Thu Oct 20 08:48:00 2022] GET /identity/v3/roles/member => generated 84 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:00.705062 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0421aab8-6307-4080-9599-996549502948 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:00.717962 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:00.743035 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3069be5d-488d-47ad-bc35-1da8624a3d75 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:00.759893 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:00.777208 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a9ceb1-c503-4cbb-aadf-d8a49796d67d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:00.777829 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a9ceb1-c503-4cbb-aadf-d8a49796d67d None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:00.778103 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a9ceb1-c503-4cbb-aadf-d8a49796d67d None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:00.778375 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a9ceb1-c503-4cbb-aadf-d8a49796d67d None admin] PATH_INFO: `/v3/domains/default` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:00.779609 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72a9ceb1-c503-4cbb-aadf-d8a49796d67d None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:00.781046 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72a9ceb1-c503-4cbb-aadf-d8a49796d67d None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:00.782928 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 103/211] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:48:00 2022] GET /identity/v3/domains/default => generated 199 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 08:48:00.783889 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff606c6-5fe5-45b9-a10d-1e5a611f73dd None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:00.784171 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff606c6-5fe5-45b9-a10d-1e5a611f73dd None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:00.784438 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff606c6-5fe5-45b9-a10d-1e5a611f73dd None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:00.785778 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 104/212] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:48:00 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:48:00.790672 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff606c6-5fe5-45b9-a10d-1e5a611f73dd None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:00.805212 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0421aab8-6307-4080-9599-996549502948 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:00.839593 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2f18274-ad68-440d-a460-497210718307 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:00.840219 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f18274-ad68-440d-a460-497210718307 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:00.840494 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f18274-ad68-440d-a460-497210718307 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:00.840755 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f18274-ad68-440d-a460-497210718307 None admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:00.841413 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2f18274-ad68-440d-a460-497210718307 None admin] RBAC: Adding query filter params (name=member) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:00.841706 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2f18274-ad68-440d-a460-497210718307 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:00.843127 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2f18274-ad68-440d-a460-497210718307 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:00.851091 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 109/213] 10.222.0.36 () {56 vars in 1101 bytes} [Thu Oct 20 08:48:00 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:00.852359 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85f2d95-22f2-45f9-972b-91905a70e055 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:00.852645 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85f2d95-22f2-45f9-972b-91905a70e055 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:00.852913 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85f2d95-22f2-45f9-972b-91905a70e055 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:00.870802 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-a85f2d95-22f2-45f9-972b-91905a70e055 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:48:00.871988 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-a85f2d95-22f2-45f9-972b-91905a70e055 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:48:00.894744 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff606c6-5fe5-45b9-a10d-1e5a611f73dd None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:00.928150 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae8f9dfa-2bf0-45a2-abc5-aab7f96039e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:00.928745 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae8f9dfa-2bf0-45a2-abc5-aab7f96039e1 None admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:00.929023 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae8f9dfa-2bf0-45a2-abc5-aab7f96039e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:00.929291 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae8f9dfa-2bf0-45a2-abc5-aab7f96039e1 None admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:00.929930 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae8f9dfa-2bf0-45a2-abc5-aab7f96039e1 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:00.931394 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae8f9dfa-2bf0-45a2-abc5-aab7f96039e1 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:00.935422 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 110/214] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:48:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 20 08:48:00.937380 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a85f2d95-22f2-45f9-972b-91905a70e055 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:00.940094 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:00.978509 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 105/215] 10.222.0.36 () {60 vars in 1132 bytes} [Thu Oct 20 08:48:00 2022] POST /identity/v3/users => generated 293 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 08:48:01.060775 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a85f2d95-22f2-45f9-972b-91905a70e055 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:01.097864 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6acbc0db-c7a3-48e3-994d-2cb4c7d8e1a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:01.098535 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6acbc0db-c7a3-48e3-994d-2cb4c7d8e1a8 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:01.098816 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6acbc0db-c7a3-48e3-994d-2cb4c7d8e1a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:01.099106 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6acbc0db-c7a3-48e3-994d-2cb4c7d8e1a8 None admin] PATH_INFO: `/v3/users/afbd7581154c4200a12f3398eaa577dd` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:01.113931 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6acbc0db-c7a3-48e3-994d-2cb4c7d8e1a8 None admin] RBAC: Authorizing `identity:get_user(user_id=afbd7581154c4200a12f3398eaa577dd)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:01.116015 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6acbc0db-c7a3-48e3-994d-2cb4c7d8e1a8 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:01.117749 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 111/216] 10.222.0.36 () {56 vars in 1177 bytes} [Thu Oct 20 08:48:00 2022] GET /identity/v3/users/afbd7581154c4200a12f3398eaa577dd => generated 293 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:01.123717 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae8f9dfa-2bf0-45a2-abc5-aab7f96039e1 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:01.174261 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:01.208626 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6acbc0db-c7a3-48e3-994d-2cb4c7d8e1a8 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:01.243352 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae8f9dfa-2bf0-45a2-abc5-aab7f96039e1 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:01.289875 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:01.290586 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff32645e-490e-4382-9399-967c4baf7ced None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:01.291197 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff32645e-490e-4382-9399-967c4baf7ced None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:01.291473 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff32645e-490e-4382-9399-967c4baf7ced None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:01.291739 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff32645e-490e-4382-9399-967c4baf7ced None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:01.292593 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff32645e-490e-4382-9399-967c4baf7ced None admin] RBAC: Adding query filter params (role.id=747fbb4dfbec43baa31ec3b7761fd4b1, user.id=afbd7581154c4200a12f3398eaa577dd, scope.system=all) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:01.292816 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff32645e-490e-4382-9399-967c4baf7ced None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:01.294267 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff32645e-490e-4382-9399-967c4baf7ced None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:01.302324 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 106/217] 10.222.0.36 () {56 vars in 1308 bytes} [Thu Oct 20 08:48:01 2022] GET /identity/v3/role_assignments?user.id=afbd7581154c4200a12f3398eaa577dd&scope.system=all&role.id=747fbb4dfbec43baa31ec3b7761fd4b1 => generated 226 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:01.339311 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6acbc0db-c7a3-48e3-994d-2cb4c7d8e1a8 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:01.390438 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-633ec2c5-4c1d-454a-b3f6-b7b93e4854b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:01.391130 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-633ec2c5-4c1d-454a-b3f6-b7b93e4854b3 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:01.391404 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-633ec2c5-4c1d-454a-b3f6-b7b93e4854b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:01.391667 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-633ec2c5-4c1d-454a-b3f6-b7b93e4854b3 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:01.404432 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-633ec2c5-4c1d-454a-b3f6-b7b93e4854b3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 20 08:48:01.409426 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-633ec2c5-4c1d-454a-b3f6-b7b93e4854b3 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 20 08:48:01.409426 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:01.409426 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:01.409426 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:01.409426 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:01.409426 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:01.409426 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:01.409426 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:01.409426 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:01.409426 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:01.409426 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:01.409426 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:01.409426 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 20 08:48:01.409426 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 20 08:48:01.409426 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 20 08:48:01.409426 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 20 08:48:01.410330 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 20 08:48:01.410330 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 20 08:48:01.410330 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 20 08:48:01.410330 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 20 08:48:01.410330 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:01.410330 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:01.410330 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:01.410330 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:01.410330 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:01.410330 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:01.410330 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:01.410330 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:01.410330 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:01.410330 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:01.410330 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:01.410330 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:01.411626 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:01.411626 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:01.411626 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:01.411626 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:01.411626 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 20 08:48:01.411626 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 20 08:48:01.411626 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 20 08:48:01.411626 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 20 08:48:01.411626 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 20 08:48:01.411626 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 20 08:48:01.411626 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 20 08:48:01.411626 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:48:01.411626 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 112/218] 10.222.0.36 () {56 vars in 1099 bytes} [Thu Oct 20 08:48:01 2022] GET /identity/v3/roles/reader => generated 84 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 20 08:48:01.417432 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff32645e-490e-4382-9399-967c4baf7ced None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:01.529630 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff32645e-490e-4382-9399-967c4baf7ced None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:01.537689 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:01.565307 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-706261e5-3f83-4e57-8847-897c6c2e6b34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:01.566200 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706261e5-3f83-4e57-8847-897c6c2e6b34 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:01.566764 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706261e5-3f83-4e57-8847-897c6c2e6b34 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:01.567205 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706261e5-3f83-4e57-8847-897c6c2e6b34 None admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:01.568084 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-706261e5-3f83-4e57-8847-897c6c2e6b34 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:01.568559 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-706261e5-3f83-4e57-8847-897c6c2e6b34 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:01.570112 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-706261e5-3f83-4e57-8847-897c6c2e6b34 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:01.579694 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 107/219] 10.222.0.36 () {56 vars in 1101 bytes} [Thu Oct 20 08:48:01 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 08:48:01.586449 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-633ec2c5-4c1d-454a-b3f6-b7b93e4854b3 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:01.692289 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-633ec2c5-4c1d-454a-b3f6-b7b93e4854b3 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:01.714711 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:01.729686 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-419cc8c1-db77-43d0-a9f0-d2e1e6274094 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:01.730319 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-419cc8c1-db77-43d0-a9f0-d2e1e6274094 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:01.730632 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-419cc8c1-db77-43d0-a9f0-d2e1e6274094 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:01.730905 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-419cc8c1-db77-43d0-a9f0-d2e1e6274094 None admin] PATH_INFO: `/v3/users/cae79f71b1364dca9beb3590abc86b3f` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:01.732032 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-419cc8c1-db77-43d0-a9f0-d2e1e6274094 None admin] RBAC: Authorizing `identity:get_user(user_id=cae79f71b1364dca9beb3590abc86b3f)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:01.733503 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-419cc8c1-db77-43d0-a9f0-d2e1e6274094 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:01.735116 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 113/220] 10.222.0.36 () {56 vars in 1177 bytes} [Thu Oct 20 08:48:01 2022] GET /identity/v3/users/cae79f71b1364dca9beb3590abc86b3f => generated 297 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:01.740809 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-706261e5-3f83-4e57-8847-897c6c2e6b34 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:01.842727 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-706261e5-3f83-4e57-8847-897c6c2e6b34 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:01.878343 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-657636b1-04de-4dae-8fcc-411572d3ba61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:01.878969 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-657636b1-04de-4dae-8fcc-411572d3ba61 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:01.879242 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-657636b1-04de-4dae-8fcc-411572d3ba61 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:01.879510 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-657636b1-04de-4dae-8fcc-411572d3ba61 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:01.915300 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:01.951047 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-657636b1-04de-4dae-8fcc-411572d3ba61 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:01.952445 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-657636b1-04de-4dae-8fcc-411572d3ba61 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:02.060818 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 108/221] 10.222.0.36 () {58 vars in 1282 bytes} [Thu Oct 20 08:48:01 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 320 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 20 08:48:02.067631 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-419cc8c1-db77-43d0-a9f0-d2e1e6274094 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:02.175961 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-419cc8c1-db77-43d0-a9f0-d2e1e6274094 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:02.212129 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f112edaf-3985-4e7f-8fcc-32ee9b393981 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:02.212718 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f112edaf-3985-4e7f-8fcc-32ee9b393981 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:02.212973 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f112edaf-3985-4e7f-8fcc-32ee9b393981 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:02.213196 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f112edaf-3985-4e7f-8fcc-32ee9b393981 None admin] PATH_INFO: `/v3/projects/a00517b9cca94187ac4aba4ec7468007` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:02.214342 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f112edaf-3985-4e7f-8fcc-32ee9b393981 None admin] RBAC: Authorizing `identity:get_project(project_id=a00517b9cca94187ac4aba4ec7468007)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:02.215825 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f112edaf-3985-4e7f-8fcc-32ee9b393981 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:02.217734 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 114/222] 10.222.0.36 () {56 vars in 1186 bytes} [Thu Oct 20 08:48:02 2022] GET /identity/v3/projects/a00517b9cca94187ac4aba4ec7468007 => generated 302 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:02.223494 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-657636b1-04de-4dae-8fcc-411572d3ba61 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:02.295172 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:02.329791 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-657636b1-04de-4dae-8fcc-411572d3ba61 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:02.364624 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-405cf85e-1da6-43f9-8836-f52eae7c5ded None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:02.366137 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405cf85e-1da6-43f9-8836-f52eae7c5ded None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:02.366664 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405cf85e-1da6-43f9-8836-f52eae7c5ded None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:02.367139 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405cf85e-1da6-43f9-8836-f52eae7c5ded None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:02.368076 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-405cf85e-1da6-43f9-8836-f52eae7c5ded None admin] RBAC: Adding query filter params (role.id=9bfe29e09c61423aa168f5f0f0cbef17, scope.project.id=a00517b9cca94187ac4aba4ec7468007, user.id=cae79f71b1364dca9beb3590abc86b3f) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:02.368612 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-405cf85e-1da6-43f9-8836-f52eae7c5ded None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:02.370276 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-405cf85e-1da6-43f9-8836-f52eae7c5ded None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:02.376854 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 109/223] 10.222.0.36 () {56 vars in 1374 bytes} [Thu Oct 20 08:48:02 2022] GET /identity/v3/role_assignments?user.id=cae79f71b1364dca9beb3590abc86b3f&scope.project.id=a00517b9cca94187ac4aba4ec7468007&role.id=9bfe29e09c61423aa168f5f0f0cbef17 => generated 608 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:03.290696 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ab1a7c-c1ed-4bab-91e9-2e78d1131429 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:03.291255 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ab1a7c-c1ed-4bab-91e9-2e78d1131429 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:03.291255 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ab1a7c-c1ed-4bab-91e9-2e78d1131429 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:03.292730 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 115/224] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:48:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:48:03.297711 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0316e0d-d5bd-4234-aa20-4ad28a0a613b None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:03.298085 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0316e0d-d5bd-4234-aa20-4ad28a0a613b None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:03.298260 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0316e0d-d5bd-4234-aa20-4ad28a0a613b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:03.316438 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-a0316e0d-d5bd-4234-aa20-4ad28a0a613b None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:48:03.317626 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-a0316e0d-d5bd-4234-aa20-4ad28a0a613b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:48:03.397771 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 110/225] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:48:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 20 08:48:03.450016 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f26c33d-5aef-46da-8be7-1a03ea7b62e5 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:03.450314 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f26c33d-5aef-46da-8be7-1a03ea7b62e5 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:03.450577 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f26c33d-5aef-46da-8be7-1a03ea7b62e5 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:03.451960 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 116/226] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:48:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:48:03.458236 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e1f1acf-6e65-428e-a69d-d85d09d9fa8c None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:03.458621 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e1f1acf-6e65-428e-a69d-d85d09d9fa8c None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:03.459382 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e1f1acf-6e65-428e-a69d-d85d09d9fa8c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:03.478328 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-0e1f1acf-6e65-428e-a69d-d85d09d9fa8c None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:48:03.479572 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-0e1f1acf-6e65-428e-a69d-d85d09d9fa8c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:48:03.537780 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:03.549190 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 111/227] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:48:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 20 08:48:03.552125 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2bd36f6-73d1-4134-9ea6-180f4be7a3b7 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:03.552442 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2bd36f6-73d1-4134-9ea6-180f4be7a3b7 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:03.552713 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2bd36f6-73d1-4134-9ea6-180f4be7a3b7 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:03.554027 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 117/228] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:48:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:48:03.559059 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a15a49a-d504-4696-8835-fad01f330622 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:03.559342 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a15a49a-d504-4696-8835-fad01f330622 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:03.559610 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a15a49a-d504-4696-8835-fad01f330622 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:03.577872 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-7a15a49a-d504-4696-8835-fad01f330622 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:48:03.579075 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-7a15a49a-d504-4696-8835-fad01f330622 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:48:03.646765 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 112/229] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:48:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 20 08:48:03.663670 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2bd36f6-73d1-4134-9ea6-180f4be7a3b7 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:03.721565 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:03.767078 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2bd36f6-73d1-4134-9ea6-180f4be7a3b7 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:03.800206 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4197904f-6611-4eb4-80a8-f30a73fdf808 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:03.800813 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4197904f-6611-4eb4-80a8-f30a73fdf808 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:03.801090 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4197904f-6611-4eb4-80a8-f30a73fdf808 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:03.801353 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4197904f-6611-4eb4-80a8-f30a73fdf808 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:03.811790 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4197904f-6611-4eb4-80a8-f30a73fdf808 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 20 08:48:03.816496 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-4197904f-6611-4eb4-80a8-f30a73fdf808 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 20 08:48:03.816496 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:03.816496 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:03.816496 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:03.816496 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:03.816496 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:03.816496 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:03.816496 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:03.816496 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:03.816496 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:03.816496 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:03.816496 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:03.816496 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 20 08:48:03.816496 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 20 08:48:03.816496 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 20 08:48:03.816496 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 20 08:48:03.817405 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 20 08:48:03.817405 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 20 08:48:03.817405 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 20 08:48:03.817405 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 20 08:48:03.817405 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:03.817405 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:03.817405 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:03.817405 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:03.817405 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:03.817405 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:03.817405 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:03.817405 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:03.817405 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:03.817405 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:03.817405 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:03.817405 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:03.818865 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:03.818865 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:03.818865 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:03.818865 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:03.818865 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 20 08:48:03.818865 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 20 08:48:03.818865 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 20 08:48:03.818865 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 20 08:48:03.818865 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 20 08:48:03.818865 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 20 08:48:03.818865 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 20 08:48:03.818865 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:48:03.818865 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 118/230] 10.222.0.36 () {56 vars in 1099 bytes} [Thu Oct 20 08:48:03 2022] GET /identity/v3/roles/reader => generated 84 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:03.819848 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a15a49a-d504-4696-8835-fad01f330622 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:03.825111 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4197904f-6611-4eb4-80a8-f30a73fdf808 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:03.927122 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a15a49a-d504-4696-8835-fad01f330622 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:03.936860 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4197904f-6611-4eb4-80a8-f30a73fdf808 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:03.951268 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:03.953186 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:03.962754 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ef99469-e7ff-426a-a066-b7f2f35ca696 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:03.963381 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ef99469-e7ff-426a-a066-b7f2f35ca696 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:03.963655 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ef99469-e7ff-426a-a066-b7f2f35ca696 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:03.963921 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ef99469-e7ff-426a-a066-b7f2f35ca696 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:03.965008 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ef99469-e7ff-426a-a066-b7f2f35ca696 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:03.966440 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ef99469-e7ff-426a-a066-b7f2f35ca696 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:03.968025 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 113/231] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:48:03 2022] GET /identity/v3/domains/default => generated 199 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:03.968784 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ef99469-e7ff-426a-a066-b7f2f35ca696 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:03.973967 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-458c2980-45a1-4ed2-b981-434604809ff6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:03.974598 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458c2980-45a1-4ed2-b981-434604809ff6 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:03.974876 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458c2980-45a1-4ed2-b981-434604809ff6 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:03.975140 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458c2980-45a1-4ed2-b981-434604809ff6 None admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:03.975795 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-458c2980-45a1-4ed2-b981-434604809ff6 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:03.976091 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-458c2980-45a1-4ed2-b981-434604809ff6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:03.977455 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-458c2980-45a1-4ed2-b981-434604809ff6 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:03.985522 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 119/232] 10.222.0.36 () {56 vars in 1101 bytes} [Thu Oct 20 08:48:03 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 08:48:03.986330 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-458c2980-45a1-4ed2-b981-434604809ff6 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:04.066552 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ef99469-e7ff-426a-a066-b7f2f35ca696 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:04.085183 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-458c2980-45a1-4ed2-b981-434604809ff6 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:04.102055 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45b871cc-1db7-46c5-adfb-4aa4e1783b7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:04.102693 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45b871cc-1db7-46c5-adfb-4aa4e1783b7e None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:04.102967 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45b871cc-1db7-46c5-adfb-4aa4e1783b7e None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:04.103233 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45b871cc-1db7-46c5-adfb-4aa4e1783b7e None admin] PATH_INFO: `/v3/roles/member` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:04.112923 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45b871cc-1db7-46c5-adfb-4aa4e1783b7e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 20 08:48:04.117703 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7f2f94b-ccf1-4a69-a67d-4374cef2fe71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:04.118166 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-45b871cc-1db7-46c5-adfb-4aa4e1783b7e None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 20 08:48:04.118166 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:04.118166 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:04.118166 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:04.118166 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:04.118166 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:04.118166 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:04.118166 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:04.118166 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:04.118166 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:04.118166 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:04.118166 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:04.118166 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 20 08:48:04.118166 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 20 08:48:04.118166 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 20 08:48:04.118166 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 20 08:48:04.119265 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 20 08:48:04.119265 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 20 08:48:04.119265 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 20 08:48:04.119265 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 20 08:48:04.119265 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:04.119265 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:04.119265 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:04.119265 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:04.119265 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:04.119265 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:04.119265 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:04.119265 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:04.119265 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:04.119265 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:04.119265 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:04.119265 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:04.125054 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:04.125054 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:04.125054 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:04.125054 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:04.125054 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 20 08:48:04.125054 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 20 08:48:04.125054 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 20 08:48:04.125054 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 20 08:48:04.125054 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 20 08:48:04.125054 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 20 08:48:04.125054 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 20 08:48:04.125054 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:04.125678 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f2f94b-ccf1-4a69-a67d-4374cef2fe71 None admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:04.125678 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f2f94b-ccf1-4a69-a67d-4374cef2fe71 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:04.125678 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f2f94b-ccf1-4a69-a67d-4374cef2fe71 None admin] PATH_INFO: `/v3/groups` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:04.125861 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 114/233] 10.222.0.36 () {56 vars in 1099 bytes} [Thu Oct 20 08:48:03 2022] GET /identity/v3/roles/member => generated 84 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:04.125936 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7f2f94b-ccf1-4a69-a67d-4374cef2fe71 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:04.126003 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45b871cc-1db7-46c5-adfb-4aa4e1783b7e None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:04.126070 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7f2f94b-ccf1-4a69-a67d-4374cef2fe71 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:04.136476 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 120/234] 10.222.0.36 () {60 vars in 1134 bytes} [Thu Oct 20 08:48:03 2022] POST /identity/v3/groups => generated 229 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 08:48:04.137278 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7f2f94b-ccf1-4a69-a67d-4374cef2fe71 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:04.181869 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:04.186119 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:04.247077 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45b871cc-1db7-46c5-adfb-4aa4e1783b7e None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:04.252646 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7f2f94b-ccf1-4a69-a67d-4374cef2fe71 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:04.287829 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aca63bf-9eae-43df-a1cc-37e527d6106b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:04.288481 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aca63bf-9eae-43df-a1cc-37e527d6106b None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:04.288753 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aca63bf-9eae-43df-a1cc-37e527d6106b None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:04.289190 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aca63bf-9eae-43df-a1cc-37e527d6106b None admin] PATH_INFO: `/v3/users/7f5a3fc6cc5d439b8f345c358d08eeaa` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:04.294002 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a0a239-1743-42d0-ad42-c7da334ed48b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:04.294683 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a0a239-1743-42d0-ad42-c7da334ed48b None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:04.295007 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a0a239-1743-42d0-ad42-c7da334ed48b None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:04.295279 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a0a239-1743-42d0-ad42-c7da334ed48b None admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:04.295964 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a0a239-1743-42d0-ad42-c7da334ed48b None admin] RBAC: Adding query filter params (name=member) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:04.296267 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a0a239-1743-42d0-ad42-c7da334ed48b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:04.297704 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a0a239-1743-42d0-ad42-c7da334ed48b None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:04.304104 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aca63bf-9eae-43df-a1cc-37e527d6106b None admin] RBAC: Authorizing `identity:get_user(user_id=7f5a3fc6cc5d439b8f345c358d08eeaa)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:04.307776 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aca63bf-9eae-43df-a1cc-37e527d6106b None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:04.307891 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 121/235] 10.222.0.36 () {56 vars in 1101 bytes} [Thu Oct 20 08:48:04 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:04.311210 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 115/236] 10.222.0.36 () {56 vars in 1177 bytes} [Thu Oct 20 08:48:04 2022] GET /identity/v3/users/7f5a3fc6cc5d439b8f345c358d08eeaa => generated 293 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:04.312409 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a0a239-1743-42d0-ad42-c7da334ed48b None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:04.314395 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aca63bf-9eae-43df-a1cc-37e527d6106b None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:04.425603 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:04.428430 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aca63bf-9eae-43df-a1cc-37e527d6106b None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:04.428823 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a0a239-1743-42d0-ad42-c7da334ed48b None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:04.438753 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:04.464708 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a170627a-2ea3-485c-93d2-d31a94eced5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:04.465307 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a170627a-2ea3-485c-93d2-d31a94eced5b None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:04.465580 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a170627a-2ea3-485c-93d2-d31a94eced5b None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:04.465835 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a170627a-2ea3-485c-93d2-d31a94eced5b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:04.466586 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a170627a-2ea3-485c-93d2-d31a94eced5b None admin] RBAC: Adding query filter params (role.id=9bfe29e09c61423aa168f5f0f0cbef17, user.id=7f5a3fc6cc5d439b8f345c358d08eeaa, scope.system=all) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:04.466704 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc97de9-1163-4efe-896f-a42e145b3863 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:04.466884 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a170627a-2ea3-485c-93d2-d31a94eced5b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:04.467330 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc97de9-1163-4efe-896f-a42e145b3863 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:04.467612 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc97de9-1163-4efe-896f-a42e145b3863 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:04.467885 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc97de9-1163-4efe-896f-a42e145b3863 None admin] PATH_INFO: `/v3/users/afbd7581154c4200a12f3398eaa577dd` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:04.468304 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a170627a-2ea3-485c-93d2-d31a94eced5b None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:04.469112 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fc97de9-1163-4efe-896f-a42e145b3863 None admin] RBAC: Authorizing `identity:get_user(user_id=afbd7581154c4200a12f3398eaa577dd)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:04.470881 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fc97de9-1163-4efe-896f-a42e145b3863 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:04.472324 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 122/237] 10.222.0.36 () {56 vars in 1177 bytes} [Thu Oct 20 08:48:04 2022] GET /identity/v3/users/afbd7581154c4200a12f3398eaa577dd => generated 293 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:04.473744 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 116/238] 10.222.0.36 () {56 vars in 1308 bytes} [Thu Oct 20 08:48:04 2022] GET /identity/v3/role_assignments?user.id=7f5a3fc6cc5d439b8f345c358d08eeaa&scope.system=all&role.id=9bfe29e09c61423aa168f5f0f0cbef17 => generated 226 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 08:48:04.477784 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc97de9-1163-4efe-896f-a42e145b3863 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:04.587972 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc97de9-1163-4efe-896f-a42e145b3863 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:04.624190 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fbc236e-a0e2-462b-ab99-a3fc6e19ea33 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:04.624863 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fbc236e-a0e2-462b-ab99-a3fc6e19ea33 None admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:04.625143 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fbc236e-a0e2-462b-ab99-a3fc6e19ea33 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:04.625417 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fbc236e-a0e2-462b-ab99-a3fc6e19ea33 None admin] PATH_INFO: `/v3/system/users/afbd7581154c4200a12f3398eaa577dd/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:04.626918 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fbc236e-a0e2-462b-ab99-a3fc6e19ea33 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=afbd7581154c4200a12f3398eaa577dd, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:04.628473 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fbc236e-a0e2-462b-ab99-a3fc6e19ea33 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:04.637930 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 123/239] 10.222.0.36 () {58 vars in 1334 bytes} [Thu Oct 20 08:48:04 2022] PUT /identity/v3/system/users/afbd7581154c4200a12f3398eaa577dd/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 08:48:06.314369 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd3ab5d2-781d-4132-83a3-64c525483a38 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:06.314920 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd3ab5d2-781d-4132-83a3-64c525483a38 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:06.314998 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd3ab5d2-781d-4132-83a3-64c525483a38 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:06.316352 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 117/240] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:48:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:48:06.321609 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60edf89b-c0ba-4946-b4fa-c3e40e2a9ffb None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:06.322031 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60edf89b-c0ba-4946-b4fa-c3e40e2a9ffb None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:06.322208 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60edf89b-c0ba-4946-b4fa-c3e40e2a9ffb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:06.341336 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-60edf89b-c0ba-4946-b4fa-c3e40e2a9ffb None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:48:06.342546 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-60edf89b-c0ba-4946-b4fa-c3e40e2a9ffb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:48:06.371613 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:06.416611 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 124/241] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:48:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 20 08:48:06.613860 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50fea4a0-df48-48eb-9ade-195450fc8557 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:06.614177 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50fea4a0-df48-48eb-9ade-195450fc8557 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:06.614419 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50fea4a0-df48-48eb-9ade-195450fc8557 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:06.616528 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 118/242] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:48:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:48:06.621382 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b8eaa7b-7341-469e-a60e-d4b874f5833c None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:06.621669 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b8eaa7b-7341-469e-a60e-d4b874f5833c None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:06.621939 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b8eaa7b-7341-469e-a60e-d4b874f5833c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:06.639963 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-0b8eaa7b-7341-469e-a60e-d4b874f5833c None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:48:06.641139 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-0b8eaa7b-7341-469e-a60e-d4b874f5833c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:48:06.692403 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50fea4a0-df48-48eb-9ade-195450fc8557 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:06.718641 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 125/243] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:48:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 20 08:48:06.781817 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce71cd3-6a64-40a3-852a-7dad98540d8a None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:06.782110 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce71cd3-6a64-40a3-852a-7dad98540d8a None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:06.782387 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce71cd3-6a64-40a3-852a-7dad98540d8a None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:06.783909 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 126/244] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:48:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 20 08:48:06.789161 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b801130-624b-40df-a80c-48454ec22f55 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:06.789444 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b801130-624b-40df-a80c-48454ec22f55 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:06.789720 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b801130-624b-40df-a80c-48454ec22f55 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:06.799318 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50fea4a0-df48-48eb-9ade-195450fc8557 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:06.807996 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-9b801130-624b-40df-a80c-48454ec22f55 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:48:06.809168 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-9b801130-624b-40df-a80c-48454ec22f55 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:48:06.838290 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ece73f52-bef0-4ba9-8fbf-289230d0bf98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:06.839690 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece73f52-bef0-4ba9-8fbf-289230d0bf98 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:06.840242 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece73f52-bef0-4ba9-8fbf-289230d0bf98 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:06.840700 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece73f52-bef0-4ba9-8fbf-289230d0bf98 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:06.841928 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ece73f52-bef0-4ba9-8fbf-289230d0bf98 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:06.843679 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ece73f52-bef0-4ba9-8fbf-289230d0bf98 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:06.845870 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 119/245] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:48:06 2022] GET /identity/v3/domains/default => generated 199 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 08:48:06.851997 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ece73f52-bef0-4ba9-8fbf-289230d0bf98 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:06.879394 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 127/246] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:48:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 20 08:48:06.895755 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:06.953638 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ece73f52-bef0-4ba9-8fbf-289230d0bf98 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:06.987130 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7099f6ec-e5c7-4e74-b7c0-503c97cf3fce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:06.987548 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7099f6ec-e5c7-4e74-b7c0-503c97cf3fce None admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:06.987825 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7099f6ec-e5c7-4e74-b7c0-503c97cf3fce None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:06.988088 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7099f6ec-e5c7-4e74-b7c0-503c97cf3fce None admin] PATH_INFO: `/v3/groups` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:06.988183 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b801130-624b-40df-a80c-48454ec22f55 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:06.988701 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7099f6ec-e5c7-4e74-b7c0-503c97cf3fce None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:06.990681 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7099f6ec-e5c7-4e74-b7c0-503c97cf3fce None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:07.010278 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 120/247] 10.222.0.36 () {60 vars in 1134 bytes} [Thu Oct 20 08:48:06 2022] POST /identity/v3/groups => generated 226 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 20 08:48:07.018338 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:07.108459 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b801130-624b-40df-a80c-48454ec22f55 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:07.146020 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-757afd2c-38e1-4fcc-9e48-06aa937d2588 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:07.147125 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757afd2c-38e1-4fcc-9e48-06aa937d2588 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:07.147400 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757afd2c-38e1-4fcc-9e48-06aa937d2588 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:07.147665 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757afd2c-38e1-4fcc-9e48-06aa937d2588 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:07.159129 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-757afd2c-38e1-4fcc-9e48-06aa937d2588 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 20 08:48:07.159316 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7099f6ec-e5c7-4e74-b7c0-503c97cf3fce None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:07.165871 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-757afd2c-38e1-4fcc-9e48-06aa937d2588 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 20 08:48:07.165871 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:07.165871 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:07.165871 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:07.165871 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:07.165871 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:07.165871 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:07.165871 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:07.165871 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:07.165871 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:07.165871 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:07.165871 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:07.165871 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 20 08:48:07.165871 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 20 08:48:07.165871 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 20 08:48:07.165871 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 20 08:48:07.166892 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 20 08:48:07.166892 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 20 08:48:07.166892 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 20 08:48:07.166892 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 20 08:48:07.166892 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:07.166892 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:07.166892 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:07.166892 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:07.166892 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:07.166892 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:07.166892 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:07.166892 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:07.166892 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:07.166892 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:07.166892 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:07.166892 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:07.171469 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:07.171469 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:07.171469 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:07.171469 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:07.171469 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 20 08:48:07.171469 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 20 08:48:07.171469 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 20 08:48:07.171469 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 20 08:48:07.171469 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 20 08:48:07.171469 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 20 08:48:07.171469 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 20 08:48:07.171469 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:48:07.171469 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 128/248] 10.222.0.36 () {56 vars in 1099 bytes} [Thu Oct 20 08:48:06 2022] GET /identity/v3/roles/reader => generated 84 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 20 08:48:07.173186 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-757afd2c-38e1-4fcc-9e48-06aa937d2588 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:07.271875 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:07.296755 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:07.299831 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7099f6ec-e5c7-4e74-b7c0-503c97cf3fce None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:07.300523 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-757afd2c-38e1-4fcc-9e48-06aa937d2588 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:07.342295 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-379041b2-dcab-42a4-8077-792baf043d2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:07.342973 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-379041b2-dcab-42a4-8077-792baf043d2f None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:07.343257 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-379041b2-dcab-42a4-8077-792baf043d2f None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:07.343528 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-379041b2-dcab-42a4-8077-792baf043d2f None admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:07.344199 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-379041b2-dcab-42a4-8077-792baf043d2f None admin] RBAC: Adding query filter params (name=reader) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:07.344503 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-379041b2-dcab-42a4-8077-792baf043d2f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:07.345914 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-379041b2-dcab-42a4-8077-792baf043d2f None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:07.348550 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8971b47-53c6-4907-916d-3f41dbaba1eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:07.349728 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8971b47-53c6-4907-916d-3f41dbaba1eb None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:07.350172 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8971b47-53c6-4907-916d-3f41dbaba1eb None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:07.350621 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8971b47-53c6-4907-916d-3f41dbaba1eb None admin] PATH_INFO: `/v3/roles/member` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:07.355247 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 129/249] 10.222.0.36 () {56 vars in 1101 bytes} [Thu Oct 20 08:48:07 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:07.360794 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-379041b2-dcab-42a4-8077-792baf043d2f None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:07.362873 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8971b47-53c6-4907-916d-3f41dbaba1eb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 20 08:48:07.368426 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-a8971b47-53c6-4907-916d-3f41dbaba1eb None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 20 08:48:07.368426 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:07.368426 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:07.368426 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:07.368426 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:07.368426 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:07.368426 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:07.368426 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:07.368426 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:07.368426 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:07.368426 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:07.368426 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:07.368426 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 20 08:48:07.368426 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 20 08:48:07.368426 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 20 08:48:07.368426 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 20 08:48:07.369345 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 20 08:48:07.369345 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 20 08:48:07.369345 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 20 08:48:07.369345 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 20 08:48:07.369345 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:07.369345 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:07.369345 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:07.369345 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:07.369345 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:07.369345 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:07.369345 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:07.369345 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:07.369345 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:07.369345 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:07.369345 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:07.369345 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:07.370156 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:07.370156 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:07.370156 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:07.370156 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:07.370156 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 20 08:48:07.370156 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 20 08:48:07.370156 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 20 08:48:07.370156 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 20 08:48:07.370156 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 20 08:48:07.370156 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 20 08:48:07.370156 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 20 08:48:07.370156 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:07.372742 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 121/250] 10.222.0.36 () {56 vars in 1099 bytes} [Thu Oct 20 08:48:07 2022] GET /identity/v3/roles/member => generated 84 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 20 08:48:07.378726 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8971b47-53c6-4907-916d-3f41dbaba1eb None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:07.503753 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-379041b2-dcab-42a4-8077-792baf043d2f None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:07.511459 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8971b47-53c6-4907-916d-3f41dbaba1eb None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:07.543706 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b775f370-37d1-4acc-b73e-380a681c226f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:07.544339 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b775f370-37d1-4acc-b73e-380a681c226f None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:07.544616 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b775f370-37d1-4acc-b73e-380a681c226f None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:07.544883 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b775f370-37d1-4acc-b73e-380a681c226f None admin] PATH_INFO: `/v3/users/7f5a3fc6cc5d439b8f345c358d08eeaa` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:07.546015 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b775f370-37d1-4acc-b73e-380a681c226f None admin] RBAC: Authorizing `identity:get_user(user_id=7f5a3fc6cc5d439b8f345c358d08eeaa)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:07.547512 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b775f370-37d1-4acc-b73e-380a681c226f None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:07.549242 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 130/251] 10.222.0.36 () {56 vars in 1177 bytes} [Thu Oct 20 08:48:07 2022] GET /identity/v3/users/7f5a3fc6cc5d439b8f345c358d08eeaa => generated 293 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 08:48:07.555644 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b775f370-37d1-4acc-b73e-380a681c226f None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:07.557338 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e7c4a96-ea8e-452e-8e32-67f8c3f49993 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:07.557953 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7c4a96-ea8e-452e-8e32-67f8c3f49993 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:07.558231 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7c4a96-ea8e-452e-8e32-67f8c3f49993 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:07.558825 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7c4a96-ea8e-452e-8e32-67f8c3f49993 None admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:07.559501 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e7c4a96-ea8e-452e-8e32-67f8c3f49993 None admin] RBAC: Adding query filter params (name=member) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:07.559802 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e7c4a96-ea8e-452e-8e32-67f8c3f49993 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:07.562215 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e7c4a96-ea8e-452e-8e32-67f8c3f49993 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:07.572909 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 122/252] 10.222.0.36 () {56 vars in 1101 bytes} [Thu Oct 20 08:48:07 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 08:48:07.577655 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:07.579970 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e7c4a96-ea8e-452e-8e32-67f8c3f49993 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:07.638216 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:07.694414 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b775f370-37d1-4acc-b73e-380a681c226f None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:07.712542 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e7c4a96-ea8e-452e-8e32-67f8c3f49993 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:07.750872 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0be429db-5206-42e7-a480-c8a22cce871e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:07.751530 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be429db-5206-42e7-a480-c8a22cce871e None admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:07.751807 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be429db-5206-42e7-a480-c8a22cce871e None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:07.752078 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be429db-5206-42e7-a480-c8a22cce871e None admin] PATH_INFO: `/v3/system/users/7f5a3fc6cc5d439b8f345c358d08eeaa/roles/9bfe29e09c61423aa168f5f0f0cbef17` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:07.754038 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0be429db-5206-42e7-a480-c8a22cce871e None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=7f5a3fc6cc5d439b8f345c358d08eeaa, role_id=9bfe29e09c61423aa168f5f0f0cbef17)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:07.755422 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0be429db-5206-42e7-a480-c8a22cce871e None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:07.758782 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2208a50c-a7ca-42d0-8a19-9a5c2a3a1570 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:07.759766 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2208a50c-a7ca-42d0-8a19-9a5c2a3a1570 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:07.760268 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2208a50c-a7ca-42d0-8a19-9a5c2a3a1570 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:07.760753 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2208a50c-a7ca-42d0-8a19-9a5c2a3a1570 None admin] PATH_INFO: `/v3/users/afbd7581154c4200a12f3398eaa577dd` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:07.762173 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2208a50c-a7ca-42d0-8a19-9a5c2a3a1570 None admin] RBAC: Authorizing `identity:get_user(user_id=afbd7581154c4200a12f3398eaa577dd)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:07.763967 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2208a50c-a7ca-42d0-8a19-9a5c2a3a1570 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:07.766192 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 123/253] 10.222.0.36 () {56 vars in 1177 bytes} [Thu Oct 20 08:48:07 2022] GET /identity/v3/users/afbd7581154c4200a12f3398eaa577dd => generated 293 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:07.772201 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2208a50c-a7ca-42d0-8a19-9a5c2a3a1570 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:07.776769 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 131/254] 10.222.0.36 () {58 vars in 1334 bytes} [Thu Oct 20 08:48:07 2022] PUT /identity/v3/system/users/7f5a3fc6cc5d439b8f345c358d08eeaa/roles/9bfe29e09c61423aa168f5f0f0cbef17 => generated 0 bytes in 222 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 08:48:07.883231 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:07.894873 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2208a50c-a7ca-42d0-8a19-9a5c2a3a1570 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:07.937323 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2900cc9e-ec43-4b5b-a582-a20a55311d1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:07.937955 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2900cc9e-ec43-4b5b-a582-a20a55311d1b None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:07.938233 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2900cc9e-ec43-4b5b-a582-a20a55311d1b None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:07.938531 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2900cc9e-ec43-4b5b-a582-a20a55311d1b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:07.939275 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2900cc9e-ec43-4b5b-a582-a20a55311d1b None admin] RBAC: Adding query filter params (role.id=747fbb4dfbec43baa31ec3b7761fd4b1, user.id=afbd7581154c4200a12f3398eaa577dd, scope.system=all) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:07.939586 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2900cc9e-ec43-4b5b-a582-a20a55311d1b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:07.941152 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2900cc9e-ec43-4b5b-a582-a20a55311d1b None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:07.961506 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 124/255] 10.222.0.36 () {56 vars in 1308 bytes} [Thu Oct 20 08:48:07 2022] GET /identity/v3/role_assignments?user.id=afbd7581154c4200a12f3398eaa577dd&scope.system=all&role.id=747fbb4dfbec43baa31ec3b7761fd4b1 => generated 510 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:09.773207 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c245990-d0c8-4532-9842-21355496de6f None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:09.773685 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c245990-d0c8-4532-9842-21355496de6f None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:09.773685 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c245990-d0c8-4532-9842-21355496de6f None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:09.775380 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 132/256] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:48:09 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:48:09.781034 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6264bd1-bab6-47b0-b408-870574e3f4c4 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:09.781454 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6264bd1-bab6-47b0-b408-870574e3f4c4 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:09.781628 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6264bd1-bab6-47b0-b408-870574e3f4c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:09.807052 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-d6264bd1-bab6-47b0-b408-870574e3f4c4 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:48:09.808271 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-d6264bd1-bab6-47b0-b408-870574e3f4c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:48:09.891905 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 125/257] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:48:09 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 20 08:48:10.213410 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e1ba816-49e5-4fc0-83b1-204d9d96c1fe None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:10.213793 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e1ba816-49e5-4fc0-83b1-204d9d96c1fe None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:10.214012 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e1ba816-49e5-4fc0-83b1-204d9d96c1fe None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:10.215728 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 133/258] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:48:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:48:10.226759 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7451b2f6-0da1-41b6-b336-ae5b6f3888ec None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:10.227067 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7451b2f6-0da1-41b6-b336-ae5b6f3888ec None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:10.227323 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7451b2f6-0da1-41b6-b336-ae5b6f3888ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:10.251809 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-7451b2f6-0da1-41b6-b336-ae5b6f3888ec None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:48:10.253023 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-7451b2f6-0da1-41b6-b336-ae5b6f3888ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:48:10.264254 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 126/259] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:48:10 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 20 08:48:10.328950 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11543a57-afe4-48eb-9d4b-ec382e532b16 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:10.329264 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11543a57-afe4-48eb-9d4b-ec382e532b16 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:10.329500 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11543a57-afe4-48eb-9d4b-ec382e532b16 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:10.331137 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 134/260] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:48:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:48:10.343871 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-711399e8-42da-497d-b641-a358e56e49bb None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:10.344153 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-711399e8-42da-497d-b641-a358e56e49bb None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:10.344427 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-711399e8-42da-497d-b641-a358e56e49bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:10.370506 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-711399e8-42da-497d-b641-a358e56e49bb None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:48:10.370506 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-711399e8-42da-497d-b641-a358e56e49bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:48:10.379387 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 127/261] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:48:10 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 20 08:48:10.492012 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11543a57-afe4-48eb-9d4b-ec382e532b16 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:10.508070 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11543a57-afe4-48eb-9d4b-ec382e532b16 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:10.512638 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d4c21b8-750e-42c2-9cfb-1ee209f01aff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:10.513258 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d4c21b8-750e-42c2-9cfb-1ee209f01aff admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:10.513533 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d4c21b8-750e-42c2-9cfb-1ee209f01aff admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:10.513802 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d4c21b8-750e-42c2-9cfb-1ee209f01aff admin admin] PATH_INFO: `/v3/roles/member` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:10.529245 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d4c21b8-750e-42c2-9cfb-1ee209f01aff admin admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 20 08:48:10.534035 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-9d4c21b8-750e-42c2-9cfb-1ee209f01aff admin admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 20 08:48:10.534035 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:10.534035 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:10.534035 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:10.534035 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:10.534035 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:10.534035 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:10.534035 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:10.534035 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:10.534035 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:10.534035 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:10.534035 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:10.534035 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 20 08:48:10.534035 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 20 08:48:10.534035 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 20 08:48:10.534035 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 20 08:48:10.534925 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 20 08:48:10.534925 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 20 08:48:10.534925 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 20 08:48:10.534925 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 20 08:48:10.534925 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:10.534925 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:10.534925 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:10.534925 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:10.534925 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:10.534925 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:10.534925 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:10.534925 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:10.534925 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:10.534925 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:10.534925 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:10.534925 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:10.536129 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:10.536129 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:10.536129 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:10.536129 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:10.536129 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 20 08:48:10.536129 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 20 08:48:10.536129 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 20 08:48:10.536129 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 20 08:48:10.536129 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 20 08:48:10.536129 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 20 08:48:10.536129 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 20 08:48:10.536129 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:48:10.536129 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 135/262] 10.222.0.36 () {56 vars in 1120 bytes} [Thu Oct 20 08:48:10 2022] GET /identity/v3/roles/member => generated 84 bytes in 44 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:10.543687 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-711399e8-42da-497d-b641-a358e56e49bb None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:10.554006 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-711399e8-42da-497d-b641-a358e56e49bb None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:10.566075 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeba1c42-224f-497f-a54d-1d636620fa98 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:10.566721 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeba1c42-224f-497f-a54d-1d636620fa98 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:10.567000 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeba1c42-224f-497f-a54d-1d636620fa98 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:10.567274 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeba1c42-224f-497f-a54d-1d636620fa98 admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:10.567977 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeba1c42-224f-497f-a54d-1d636620fa98 admin admin] RBAC: Adding query filter params (name=member) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:10.568281 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeba1c42-224f-497f-a54d-1d636620fa98 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:10.570058 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: WARNING py.warnings [None req-eeba1c42-224f-497f-a54d-1d636620fa98 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 20 08:48:10.570058 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: warnings.warn(msg) Oct 20 08:48:10.570058 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: Oct 20 08:48:10.571075 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeba1c42-224f-497f-a54d-1d636620fa98 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:10.580093 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 128/263] 10.222.0.36 () {56 vars in 1122 bytes} [Thu Oct 20 08:48:10 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:10.586699 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d4c21b8-750e-42c2-9cfb-1ee209f01aff admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:10.600307 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d4c21b8-750e-42c2-9cfb-1ee209f01aff admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:10.610085 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef0eb4c7-a4a3-4d9a-a393-8744be478d3f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:10.610773 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef0eb4c7-a4a3-4d9a-a393-8744be478d3f admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:10.611053 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef0eb4c7-a4a3-4d9a-a393-8744be478d3f admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:10.611324 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef0eb4c7-a4a3-4d9a-a393-8744be478d3f admin admin] PATH_INFO: `/v3/groups/96d598164a48402eb014d5f9cbad95f8` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:10.624245 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef0eb4c7-a4a3-4d9a-a393-8744be478d3f admin admin] RBAC: Authorizing `identity:get_group(group_id=96d598164a48402eb014d5f9cbad95f8)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:10.626107 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: WARNING py.warnings [None req-ef0eb4c7-a4a3-4d9a-a393-8744be478d3f admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:get_group": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.group.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 20 08:48:10.626107 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: warnings.warn(msg) Oct 20 08:48:10.626107 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: Oct 20 08:48:10.627399 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef0eb4c7-a4a3-4d9a-a393-8744be478d3f admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:10.629077 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 136/264] 10.222.0.36 () {56 vars in 1201 bytes} [Thu Oct 20 08:48:10 2022] GET /identity/v3/groups/96d598164a48402eb014d5f9cbad95f8 => generated 226 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:10.635455 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeba1c42-224f-497f-a54d-1d636620fa98 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:10.651760 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeba1c42-224f-497f-a54d-1d636620fa98 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:10.656184 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de4605cb-5493-4281-9b5b-682b7a878ec2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:10.656810 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de4605cb-5493-4281-9b5b-682b7a878ec2 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:10.657088 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de4605cb-5493-4281-9b5b-682b7a878ec2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:10.657364 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de4605cb-5493-4281-9b5b-682b7a878ec2 admin admin] PATH_INFO: `/v3/projects/fcebd94752274a34a2068dd59f6fe700` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:10.658501 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de4605cb-5493-4281-9b5b-682b7a878ec2 admin admin] RBAC: Authorizing `identity:get_project(project_id=fcebd94752274a34a2068dd59f6fe700)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:10.660610 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de4605cb-5493-4281-9b5b-682b7a878ec2 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:10.662323 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 129/265] 10.222.0.36 () {56 vars in 1207 bytes} [Thu Oct 20 08:48:10 2022] GET /identity/v3/projects/fcebd94752274a34a2068dd59f6fe700 => generated 298 bytes in 27 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:10.668221 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef0eb4c7-a4a3-4d9a-a393-8744be478d3f admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:10.683640 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef0eb4c7-a4a3-4d9a-a393-8744be478d3f admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:10.689822 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-126ccad6-6f7c-4b84-ae88-c68256ea8633 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:10.690537 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-126ccad6-6f7c-4b84-ae88-c68256ea8633 admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:10.690789 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-126ccad6-6f7c-4b84-ae88-c68256ea8633 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:10.691061 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-126ccad6-6f7c-4b84-ae88-c68256ea8633 admin admin] PATH_INFO: `/v3/projects/fcebd94752274a34a2068dd59f6fe700/groups/96d598164a48402eb014d5f9cbad95f8/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:10.693871 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-126ccad6-6f7c-4b84-ae88-c68256ea8633 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fcebd94752274a34a2068dd59f6fe700, group_id=96d598164a48402eb014d5f9cbad95f8, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:10.694982 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: WARNING py.warnings [None req-126ccad6-6f7c-4b84-ae88-c68256ea8633 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 20 08:48:10.694982 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: warnings.warn(msg) Oct 20 08:48:10.694982 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: Oct 20 08:48:10.697024 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-126ccad6-6f7c-4b84-ae88-c68256ea8633 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:10.701150 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de4605cb-5493-4281-9b5b-682b7a878ec2 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:10.720832 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de4605cb-5493-4281-9b5b-682b7a878ec2 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:10.730617 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 137/266] 10.222.0.36 () {58 vars in 1463 bytes} [Thu Oct 20 08:48:10 2022] PUT /identity/v3/projects/fcebd94752274a34a2068dd59f6fe700/groups/96d598164a48402eb014d5f9cbad95f8/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 08:48:10.814034 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf9b639d-75de-42aa-8bc0-e42a33a61a74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:10.814678 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf9b639d-75de-42aa-8bc0-e42a33a61a74 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:10.814911 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf9b639d-75de-42aa-8bc0-e42a33a61a74 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:10.815141 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf9b639d-75de-42aa-8bc0-e42a33a61a74 admin admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:10.833787 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf9b639d-75de-42aa-8bc0-e42a33a61a74 admin admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 20 08:48:10.839354 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-bf9b639d-75de-42aa-8bc0-e42a33a61a74 admin admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 20 08:48:10.839354 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:10.839354 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:10.839354 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:10.839354 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:10.839354 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:10.839354 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:10.839354 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:10.839354 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:10.839354 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:10.839354 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:10.839354 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:10.839354 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 20 08:48:10.839354 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 20 08:48:10.839354 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 20 08:48:10.839354 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 20 08:48:10.841976 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 20 08:48:10.841976 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 20 08:48:10.841976 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 20 08:48:10.841976 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 20 08:48:10.841976 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:10.841976 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:10.841976 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:10.841976 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:10.841976 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:10.841976 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:10.841976 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:10.841976 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:10.841976 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:10.841976 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:10.841976 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:10.841976 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:10.843937 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:10.843937 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:10.843937 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:10.843937 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:10.843937 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 20 08:48:10.843937 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 20 08:48:10.843937 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 20 08:48:10.843937 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 20 08:48:10.843937 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 20 08:48:10.843937 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 20 08:48:10.843937 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 20 08:48:10.843937 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:10.843937 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 130/267] 10.222.0.36 () {56 vars in 1135 bytes} [Thu Oct 20 08:48:10 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 140 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 20 08:48:10.852273 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-126ccad6-6f7c-4b84-ae88-c68256ea8633 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:10.871807 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-126ccad6-6f7c-4b84-ae88-c68256ea8633 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:10.880205 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1190b97-9e85-46de-b8b6-780b0ab1280a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:10.881399 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1190b97-9e85-46de-b8b6-780b0ab1280a admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:10.881675 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1190b97-9e85-46de-b8b6-780b0ab1280a admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:10.881999 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1190b97-9e85-46de-b8b6-780b0ab1280a admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:10.882794 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1190b97-9e85-46de-b8b6-780b0ab1280a admin admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:10.883095 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1190b97-9e85-46de-b8b6-780b0ab1280a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:10.884175 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: WARNING py.warnings [None req-f1190b97-9e85-46de-b8b6-780b0ab1280a admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 20 08:48:10.884175 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: warnings.warn(msg) Oct 20 08:48:10.884175 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: Oct 20 08:48:10.885093 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1190b97-9e85-46de-b8b6-780b0ab1280a admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:10.900658 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 138/268] 10.222.0.36 () {56 vars in 1132 bytes} [Thu Oct 20 08:48:10 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:10.906712 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf9b639d-75de-42aa-8bc0-e42a33a61a74 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:10.920118 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf9b639d-75de-42aa-8bc0-e42a33a61a74 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:10.931510 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3963173-67ea-4baf-b0aa-9dd5f548e1ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:10.932161 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3963173-67ea-4baf-b0aa-9dd5f548e1ba admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:10.932438 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3963173-67ea-4baf-b0aa-9dd5f548e1ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:10.932709 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3963173-67ea-4baf-b0aa-9dd5f548e1ba admin admin] PATH_INFO: `/v3/groups/96d598164a48402eb014d5f9cbad95f8` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:10.939357 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3963173-67ea-4baf-b0aa-9dd5f548e1ba admin admin] RBAC: Authorizing `identity:get_group(group_id=96d598164a48402eb014d5f9cbad95f8)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:10.940404 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: WARNING py.warnings [None req-f3963173-67ea-4baf-b0aa-9dd5f548e1ba admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:get_group": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.group.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 20 08:48:10.940404 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: warnings.warn(msg) Oct 20 08:48:10.940404 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: Oct 20 08:48:10.941646 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3963173-67ea-4baf-b0aa-9dd5f548e1ba admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:10.944348 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 131/269] 10.222.0.36 () {56 vars in 1201 bytes} [Thu Oct 20 08:48:10 2022] GET /identity/v3/groups/96d598164a48402eb014d5f9cbad95f8 => generated 226 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:10.951230 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1190b97-9e85-46de-b8b6-780b0ab1280a admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:10.968113 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1190b97-9e85-46de-b8b6-780b0ab1280a admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:10.968869 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3963173-67ea-4baf-b0aa-9dd5f548e1ba admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:10.975529 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f06bbdd-d3cc-432a-b4d3-77d0c259cdb4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:10.976178 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f06bbdd-d3cc-432a-b4d3-77d0c259cdb4 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:10.976762 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:10.977720 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f06bbdd-d3cc-432a-b4d3-77d0c259cdb4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:10.978228 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f06bbdd-d3cc-432a-b4d3-77d0c259cdb4 admin admin] PATH_INFO: `/v3/projects/fcebd94752274a34a2068dd59f6fe700` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:10.980272 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f06bbdd-d3cc-432a-b4d3-77d0c259cdb4 admin admin] RBAC: Authorizing `identity:get_project(project_id=fcebd94752274a34a2068dd59f6fe700)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:10.985188 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3963173-67ea-4baf-b0aa-9dd5f548e1ba admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:10.986018 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f06bbdd-d3cc-432a-b4d3-77d0c259cdb4 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:10.988757 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 139/270] 10.222.0.36 () {56 vars in 1207 bytes} [Thu Oct 20 08:48:10 2022] GET /identity/v3/projects/fcebd94752274a34a2068dd59f6fe700 => generated 298 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:10.997016 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f06bbdd-d3cc-432a-b4d3-77d0c259cdb4 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:10.998682 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-639a339c-fc2a-4f05-8e02-79ffd6d07d95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:11.001436 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-639a339c-fc2a-4f05-8e02-79ffd6d07d95 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:11.002753 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-639a339c-fc2a-4f05-8e02-79ffd6d07d95 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:11.003170 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-639a339c-fc2a-4f05-8e02-79ffd6d07d95 admin admin] PATH_INFO: `/v3/roles/member` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:11.014037 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f06bbdd-d3cc-432a-b4d3-77d0c259cdb4 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:11.019233 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5b20a56-3b56-42b8-a211-e4afadafb763 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:11.020118 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b20a56-3b56-42b8-a211-e4afadafb763 admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:11.020584 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b20a56-3b56-42b8-a211-e4afadafb763 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:11.021118 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b20a56-3b56-42b8-a211-e4afadafb763 admin admin] PATH_INFO: `/v3/projects/fcebd94752274a34a2068dd59f6fe700/groups/96d598164a48402eb014d5f9cbad95f8/roles/b3f8d63bf4c4488288e57693c6e9804c` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:11.023048 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-639a339c-fc2a-4f05-8e02-79ffd6d07d95 admin admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 20 08:48:11.024295 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5b20a56-3b56-42b8-a211-e4afadafb763 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fcebd94752274a34a2068dd59f6fe700, group_id=96d598164a48402eb014d5f9cbad95f8, role_id=b3f8d63bf4c4488288e57693c6e9804c)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:11.027278 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5b20a56-3b56-42b8-a211-e4afadafb763 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:11.029430 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-639a339c-fc2a-4f05-8e02-79ffd6d07d95 admin admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 20 08:48:11.029430 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:11.029430 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:11.029430 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:11.029430 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:11.029430 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:11.029430 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:11.029430 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:11.029430 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:11.029430 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:11.029430 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:11.029430 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:11.029430 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 20 08:48:11.029430 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 20 08:48:11.029430 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 20 08:48:11.029430 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 20 08:48:11.030528 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 20 08:48:11.030528 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 20 08:48:11.030528 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 20 08:48:11.030528 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 20 08:48:11.030528 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:11.030528 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:11.030528 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:11.030528 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:11.030528 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:11.030528 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:11.030528 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:11.030528 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:11.030528 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:11.030528 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:11.030528 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:11.030528 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:11.031511 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:11.031511 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:11.031511 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:11.031511 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:11.031511 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 20 08:48:11.031511 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 20 08:48:11.031511 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 20 08:48:11.031511 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 20 08:48:11.031511 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 20 08:48:11.031511 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 20 08:48:11.031511 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 20 08:48:11.031511 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:11.033430 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 132/271] 10.222.0.36 () {56 vars in 1120 bytes} [Thu Oct 20 08:48:10 2022] GET /identity/v3/roles/member => generated 84 bytes in 65 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 20 08:48:11.035115 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39de8123-32c5-4c4f-bd11-0bda49b56f92 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:11.035115 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39de8123-32c5-4c4f-bd11-0bda49b56f92 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:11.035536 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39de8123-32c5-4c4f-bd11-0bda49b56f92 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:11.037104 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 133/272] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:48:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:48:11.041203 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39de8123-32c5-4c4f-bd11-0bda49b56f92 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:11.051527 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 140/273] 10.222.0.36 () {58 vars in 1463 bytes} [Thu Oct 20 08:48:10 2022] PUT /identity/v3/projects/fcebd94752274a34a2068dd59f6fe700/groups/96d598164a48402eb014d5f9cbad95f8/roles/b3f8d63bf4c4488288e57693c6e9804c => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 08:48:11.053274 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f2e99d-7144-437b-a5c2-71574070b75e None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:11.053861 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f2e99d-7144-437b-a5c2-71574070b75e None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:11.054330 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f2e99d-7144-437b-a5c2-71574070b75e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:11.090398 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-d2f2e99d-7144-437b-a5c2-71574070b75e None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:48:11.092202 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-d2f2e99d-7144-437b-a5c2-71574070b75e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:48:11.123577 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:11.164268 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39de8123-32c5-4c4f-bd11-0bda49b56f92 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:11.171964 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-672bddb7-e078-4197-bcec-373ad0463246 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:11.172583 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-672bddb7-e078-4197-bcec-373ad0463246 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:11.172854 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-672bddb7-e078-4197-bcec-373ad0463246 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:11.173134 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-672bddb7-e078-4197-bcec-373ad0463246 admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:11.173782 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-672bddb7-e078-4197-bcec-373ad0463246 admin admin] RBAC: Adding query filter params (name=member) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:11.174078 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-672bddb7-e078-4197-bcec-373ad0463246 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:11.175804 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-672bddb7-e078-4197-bcec-373ad0463246 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:11.187005 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 134/274] 10.222.0.36 () {56 vars in 1122 bytes} [Thu Oct 20 08:48:11 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:11.197244 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-672bddb7-e078-4197-bcec-373ad0463246 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:11.217474 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-672bddb7-e078-4197-bcec-373ad0463246 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:11.231339 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abec5467-2590-4e47-a7f4-d4931a12f6f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:11.231994 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abec5467-2590-4e47-a7f4-d4931a12f6f4 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:11.232274 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abec5467-2590-4e47-a7f4-d4931a12f6f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:11.232544 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abec5467-2590-4e47-a7f4-d4931a12f6f4 admin admin] PATH_INFO: `/v3/groups/96d598164a48402eb014d5f9cbad95f8` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:11.234502 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abec5467-2590-4e47-a7f4-d4931a12f6f4 admin admin] RBAC: Authorizing `identity:get_group(group_id=96d598164a48402eb014d5f9cbad95f8)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:11.236511 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abec5467-2590-4e47-a7f4-d4931a12f6f4 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:11.238529 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 135/275] 10.222.0.36 () {56 vars in 1201 bytes} [Thu Oct 20 08:48:11 2022] GET /identity/v3/groups/96d598164a48402eb014d5f9cbad95f8 => generated 226 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:11.244311 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abec5467-2590-4e47-a7f4-d4931a12f6f4 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:11.256521 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 141/276] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:48:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 204 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 20 08:48:11.260643 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abec5467-2590-4e47-a7f4-d4931a12f6f4 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:11.270637 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1005f847-d01f-4fb7-98d9-aad5d4845371 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:11.271878 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1005f847-d01f-4fb7-98d9-aad5d4845371 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:11.272159 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1005f847-d01f-4fb7-98d9-aad5d4845371 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:11.272503 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1005f847-d01f-4fb7-98d9-aad5d4845371 admin admin] PATH_INFO: `/v3/projects/a00517b9cca94187ac4aba4ec7468007` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:11.275321 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1005f847-d01f-4fb7-98d9-aad5d4845371 admin admin] RBAC: Authorizing `identity:get_project(project_id=a00517b9cca94187ac4aba4ec7468007)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:11.277777 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1005f847-d01f-4fb7-98d9-aad5d4845371 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:11.280032 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 136/277] 10.222.0.36 () {56 vars in 1207 bytes} [Thu Oct 20 08:48:11 2022] GET /identity/v3/projects/a00517b9cca94187ac4aba4ec7468007 => generated 302 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 08:48:11.285706 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2f2e99d-7144-437b-a5c2-71574070b75e None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:11.304667 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2f2e99d-7144-437b-a5c2-71574070b75e None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:11.319350 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdaaa8b1-3172-4e63-bfb0-1fac0503e020 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:11.320040 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdaaa8b1-3172-4e63-bfb0-1fac0503e020 admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:11.320316 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdaaa8b1-3172-4e63-bfb0-1fac0503e020 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:11.320582 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdaaa8b1-3172-4e63-bfb0-1fac0503e020 admin admin] PATH_INFO: `/v3/projects/a00517b9cca94187ac4aba4ec7468007/groups/96d598164a48402eb014d5f9cbad95f8/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:11.322696 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdaaa8b1-3172-4e63-bfb0-1fac0503e020 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a00517b9cca94187ac4aba4ec7468007, group_id=96d598164a48402eb014d5f9cbad95f8, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:11.325428 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdaaa8b1-3172-4e63-bfb0-1fac0503e020 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:11.350510 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 142/278] 10.222.0.36 () {58 vars in 1463 bytes} [Thu Oct 20 08:48:11 2022] PUT /identity/v3/projects/a00517b9cca94187ac4aba4ec7468007/groups/96d598164a48402eb014d5f9cbad95f8/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 08:48:11.533482 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60fa4279-8a21-4f6c-a6bc-ec321381d30e None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:11.534153 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60fa4279-8a21-4f6c-a6bc-ec321381d30e None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:11.534658 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60fa4279-8a21-4f6c-a6bc-ec321381d30e None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:11.536222 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 137/279] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:48:11 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:48:11.542398 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-709a1812-1f1e-4a27-9f83-7eec26b09706 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:11.543084 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-709a1812-1f1e-4a27-9f83-7eec26b09706 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:11.543556 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-709a1812-1f1e-4a27-9f83-7eec26b09706 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:11.567477 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-709a1812-1f1e-4a27-9f83-7eec26b09706 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:48:11.569048 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-709a1812-1f1e-4a27-9f83-7eec26b09706 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:48:11.654809 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 143/280] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:48:11 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 20 08:48:11.669005 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60fa4279-8a21-4f6c-a6bc-ec321381d30e None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:11.811892 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60fa4279-8a21-4f6c-a6bc-ec321381d30e None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:11.843185 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:11.854270 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1c65bbb-e6b5-4bb7-8780-826583432ff2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:11.855303 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c65bbb-e6b5-4bb7-8780-826583432ff2 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:11.855795 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c65bbb-e6b5-4bb7-8780-826583432ff2 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:11.856277 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c65bbb-e6b5-4bb7-8780-826583432ff2 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:11.870988 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1c65bbb-e6b5-4bb7-8780-826583432ff2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 20 08:48:11.876317 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-a1c65bbb-e6b5-4bb7-8780-826583432ff2 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 20 08:48:11.876317 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:11.876317 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:11.876317 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:11.876317 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:11.876317 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:11.876317 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:11.876317 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:11.876317 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:11.876317 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:11.876317 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:11.876317 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:11.876317 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 20 08:48:11.876317 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 20 08:48:11.876317 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 20 08:48:11.876317 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 20 08:48:11.877227 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 20 08:48:11.877227 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 20 08:48:11.877227 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 20 08:48:11.877227 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 20 08:48:11.877227 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:11.877227 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:11.877227 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:11.877227 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:11.877227 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:11.877227 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:11.877227 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:11.877227 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:11.877227 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:11.877227 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:11.877227 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:11.877227 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:11.878030 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:11.878030 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:11.878030 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:11.878030 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:11.878030 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 20 08:48:11.878030 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 20 08:48:11.878030 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 20 08:48:11.878030 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 20 08:48:11.878030 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 20 08:48:11.878030 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 20 08:48:11.878030 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 20 08:48:11.878030 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:11.881254 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 138/281] 10.222.0.36 () {56 vars in 1099 bytes} [Thu Oct 20 08:48:11 2022] GET /identity/v3/roles/reader => generated 84 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:11.887326 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-709a1812-1f1e-4a27-9f83-7eec26b09706 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:11.989692 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:12.018717 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-709a1812-1f1e-4a27-9f83-7eec26b09706 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:12.067409 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da10800f-5b37-4ab5-b484-b773ff3a6248 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:12.068284 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da10800f-5b37-4ab5-b484-b773ff3a6248 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:12.068568 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da10800f-5b37-4ab5-b484-b773ff3a6248 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:12.068838 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da10800f-5b37-4ab5-b484-b773ff3a6248 None admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:12.069626 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da10800f-5b37-4ab5-b484-b773ff3a6248 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:12.069927 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da10800f-5b37-4ab5-b484-b773ff3a6248 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:12.071341 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da10800f-5b37-4ab5-b484-b773ff3a6248 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:12.088581 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 144/282] 10.222.0.36 () {56 vars in 1101 bytes} [Thu Oct 20 08:48:11 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:12.094802 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1c65bbb-e6b5-4bb7-8780-826583432ff2 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:12.198258 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da10800f-5b37-4ab5-b484-b773ff3a6248 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:12.216239 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da10800f-5b37-4ab5-b484-b773ff3a6248 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:12.223255 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e07f80b-3aa3-4640-a5ea-91887388b1d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:12.224734 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e07f80b-3aa3-4640-a5ea-91887388b1d1 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:12.225248 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e07f80b-3aa3-4640-a5ea-91887388b1d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:12.225712 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e07f80b-3aa3-4640-a5ea-91887388b1d1 admin admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:12.250542 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e07f80b-3aa3-4640-a5ea-91887388b1d1 admin admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 20 08:48:12.258107 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-0e07f80b-3aa3-4640-a5ea-91887388b1d1 admin admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 20 08:48:12.258107 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:12.258107 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:12.258107 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:12.258107 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:12.258107 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:12.258107 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:12.258107 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:12.258107 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:12.258107 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:12.258107 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:12.258107 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:12.258107 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 20 08:48:12.258107 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 20 08:48:12.258107 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 20 08:48:12.258107 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 20 08:48:12.259028 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 20 08:48:12.259028 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 20 08:48:12.259028 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 20 08:48:12.259028 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 20 08:48:12.259028 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:12.259028 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:12.259028 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:12.259028 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:12.259028 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:12.259028 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:12.259028 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:12.259028 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:12.259028 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:12.259028 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:12.259028 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:12.259028 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:12.259859 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:12.259859 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:12.259859 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:12.259859 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:12.259859 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 20 08:48:12.259859 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 20 08:48:12.259859 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 20 08:48:12.259859 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 20 08:48:12.259859 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 20 08:48:12.259859 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 20 08:48:12.259859 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 20 08:48:12.259859 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:48:12.263831 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 145/283] 10.222.0.36 () {56 vars in 1135 bytes} [Thu Oct 20 08:48:12 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 66 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:12.281222 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1c65bbb-e6b5-4bb7-8780-826583432ff2 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:12.288869 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e07f80b-3aa3-4640-a5ea-91887388b1d1 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:12.312875 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e07f80b-3aa3-4640-a5ea-91887388b1d1 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:12.320457 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6df3ad0-d3ea-45f7-873d-bcd957774616 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:12.321383 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6df3ad0-d3ea-45f7-873d-bcd957774616 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:12.321860 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6df3ad0-d3ea-45f7-873d-bcd957774616 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:12.322346 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6df3ad0-d3ea-45f7-873d-bcd957774616 None admin] PATH_INFO: `/v3/users/7f5a3fc6cc5d439b8f345c358d08eeaa` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:12.323746 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6df3ad0-d3ea-45f7-873d-bcd957774616 None admin] RBAC: Authorizing `identity:get_user(user_id=7f5a3fc6cc5d439b8f345c358d08eeaa)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:12.323975 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db6ce53b-fd4a-4f17-8e0b-40194d8050d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:12.324576 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db6ce53b-fd4a-4f17-8e0b-40194d8050d6 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:12.324854 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db6ce53b-fd4a-4f17-8e0b-40194d8050d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:12.325116 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db6ce53b-fd4a-4f17-8e0b-40194d8050d6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:12.325780 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db6ce53b-fd4a-4f17-8e0b-40194d8050d6 admin admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:12.326074 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db6ce53b-fd4a-4f17-8e0b-40194d8050d6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:12.326722 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6df3ad0-d3ea-45f7-873d-bcd957774616 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:12.327712 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db6ce53b-fd4a-4f17-8e0b-40194d8050d6 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:12.328734 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 139/284] 10.222.0.36 () {56 vars in 1177 bytes} [Thu Oct 20 08:48:12 2022] GET /identity/v3/users/7f5a3fc6cc5d439b8f345c358d08eeaa => generated 293 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:12.335120 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6df3ad0-d3ea-45f7-873d-bcd957774616 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:12.343072 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 146/285] 10.222.0.36 () {56 vars in 1132 bytes} [Thu Oct 20 08:48:12 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 08:48:12.381192 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db6ce53b-fd4a-4f17-8e0b-40194d8050d6 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:12.404592 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db6ce53b-fd4a-4f17-8e0b-40194d8050d6 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:12.409962 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:12.418284 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b7e035a-a977-4df0-b18f-65c6df95585c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:12.418951 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b7e035a-a977-4df0-b18f-65c6df95585c admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:12.419225 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b7e035a-a977-4df0-b18f-65c6df95585c admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:12.419491 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b7e035a-a977-4df0-b18f-65c6df95585c admin admin] PATH_INFO: `/v3/groups/96d598164a48402eb014d5f9cbad95f8` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:12.420598 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b7e035a-a977-4df0-b18f-65c6df95585c admin admin] RBAC: Authorizing `identity:get_group(group_id=96d598164a48402eb014d5f9cbad95f8)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:12.422541 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b7e035a-a977-4df0-b18f-65c6df95585c admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:12.430736 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 147/286] 10.222.0.36 () {56 vars in 1201 bytes} [Thu Oct 20 08:48:12 2022] GET /identity/v3/groups/96d598164a48402eb014d5f9cbad95f8 => generated 226 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:12.447493 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b7e035a-a977-4df0-b18f-65c6df95585c admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:12.464206 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b7e035a-a977-4df0-b18f-65c6df95585c admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:12.467354 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6df3ad0-d3ea-45f7-873d-bcd957774616 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:12.484071 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0bf9f82-4663-4dee-b187-39de4b683c92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:12.484713 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0bf9f82-4663-4dee-b187-39de4b683c92 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:12.484987 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0bf9f82-4663-4dee-b187-39de4b683c92 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:12.485256 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0bf9f82-4663-4dee-b187-39de4b683c92 admin admin] PATH_INFO: `/v3/projects/a00517b9cca94187ac4aba4ec7468007` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:12.486368 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0bf9f82-4663-4dee-b187-39de4b683c92 admin admin] RBAC: Authorizing `identity:get_project(project_id=a00517b9cca94187ac4aba4ec7468007)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:12.488453 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0bf9f82-4663-4dee-b187-39de4b683c92 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:12.491248 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 148/287] 10.222.0.36 () {56 vars in 1207 bytes} [Thu Oct 20 08:48:12 2022] GET /identity/v3/projects/a00517b9cca94187ac4aba4ec7468007 => generated 302 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 08:48:12.509560 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0bf9f82-4663-4dee-b187-39de4b683c92 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:12.516520 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-972a613c-6aae-44dc-b7c1-5ff4eda9cb4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:12.518797 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972a613c-6aae-44dc-b7c1-5ff4eda9cb4e None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:12.519298 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972a613c-6aae-44dc-b7c1-5ff4eda9cb4e None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:12.519813 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972a613c-6aae-44dc-b7c1-5ff4eda9cb4e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:12.523213 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-972a613c-6aae-44dc-b7c1-5ff4eda9cb4e None admin] RBAC: Adding query filter params (role.id=9bfe29e09c61423aa168f5f0f0cbef17, user.id=7f5a3fc6cc5d439b8f345c358d08eeaa, scope.system=all) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:12.523796 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-972a613c-6aae-44dc-b7c1-5ff4eda9cb4e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:12.527620 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-972a613c-6aae-44dc-b7c1-5ff4eda9cb4e None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:12.534449 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0bf9f82-4663-4dee-b187-39de4b683c92 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:12.545591 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c74403d-833b-460b-be2e-261193717da5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:12.548469 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c74403d-833b-460b-be2e-261193717da5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:12.550182 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c74403d-833b-460b-be2e-261193717da5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:12.550747 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c74403d-833b-460b-be2e-261193717da5 admin admin] PATH_INFO: `/v3/projects/a00517b9cca94187ac4aba4ec7468007/groups/96d598164a48402eb014d5f9cbad95f8/roles/b3f8d63bf4c4488288e57693c6e9804c` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:12.554830 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c74403d-833b-460b-be2e-261193717da5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a00517b9cca94187ac4aba4ec7468007, group_id=96d598164a48402eb014d5f9cbad95f8, role_id=b3f8d63bf4c4488288e57693c6e9804c)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:12.558031 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 140/288] 10.222.0.36 () {56 vars in 1308 bytes} [Thu Oct 20 08:48:12 2022] GET /identity/v3/role_assignments?user.id=7f5a3fc6cc5d439b8f345c358d08eeaa&scope.system=all&role.id=9bfe29e09c61423aa168f5f0f0cbef17 => generated 510 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:12.563857 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c45d7e82-55a3-42f5-a248-f164191f7503 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:12.564457 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c45d7e82-55a3-42f5-a248-f164191f7503 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:12.564952 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c45d7e82-55a3-42f5-a248-f164191f7503 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:12.566859 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 141/289] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:48:12 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:48:12.567963 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c74403d-833b-460b-be2e-261193717da5 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:12.577947 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d417da-0f57-49fe-a14e-d3f32f0fd27d None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:12.578585 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d417da-0f57-49fe-a14e-d3f32f0fd27d None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:12.579061 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d417da-0f57-49fe-a14e-d3f32f0fd27d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:12.594348 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 149/290] 10.222.0.36 () {58 vars in 1463 bytes} [Thu Oct 20 08:48:12 2022] PUT /identity/v3/projects/a00517b9cca94187ac4aba4ec7468007/groups/96d598164a48402eb014d5f9cbad95f8/roles/b3f8d63bf4c4488288e57693c6e9804c => generated 0 bytes in 85 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 08:48:12.616368 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-c9d417da-0f57-49fe-a14e-d3f32f0fd27d None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:48:12.618090 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-c9d417da-0f57-49fe-a14e-d3f32f0fd27d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:48:12.705586 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 142/291] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:48:12 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 384 bytes (2 switches on core 0) Oct 20 08:48:13.138175 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c74403d-833b-460b-be2e-261193717da5 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:13.151566 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c74403d-833b-460b-be2e-261193717da5 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:13.160317 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebfc017a-8bbb-4f3a-a0eb-9c267b313501 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:13.160942 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebfc017a-8bbb-4f3a-a0eb-9c267b313501 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:13.161221 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebfc017a-8bbb-4f3a-a0eb-9c267b313501 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:13.161494 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebfc017a-8bbb-4f3a-a0eb-9c267b313501 admin admin] PATH_INFO: `/v3/roles/admin` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:13.180908 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebfc017a-8bbb-4f3a-a0eb-9c267b313501 admin admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 20 08:48:13.185748 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-ebfc017a-8bbb-4f3a-a0eb-9c267b313501 admin admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 20 08:48:13.185748 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:13.185748 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:13.185748 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:13.185748 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:13.185748 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:13.185748 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:13.185748 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:13.185748 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:13.185748 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:13.185748 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:13.185748 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:13.185748 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 20 08:48:13.185748 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 20 08:48:13.185748 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 20 08:48:13.185748 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 20 08:48:13.186658 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 20 08:48:13.186658 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 20 08:48:13.186658 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 20 08:48:13.186658 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 20 08:48:13.186658 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:13.186658 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:13.186658 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:13.186658 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:13.186658 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:13.186658 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:13.186658 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:13.186658 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:13.186658 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:13.186658 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:13.186658 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:13.186658 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:13.187472 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:13.187472 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:13.187472 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:13.187472 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:13.187472 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 20 08:48:13.187472 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 20 08:48:13.187472 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 20 08:48:13.187472 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 20 08:48:13.187472 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 20 08:48:13.187472 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 20 08:48:13.187472 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 20 08:48:13.187472 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:48:13.187472 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 150/292] 10.222.0.36 () {56 vars in 1117 bytes} [Thu Oct 20 08:48:13 2022] GET /identity/v3/roles/admin => generated 83 bytes in 49 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:13.192979 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9d417da-0f57-49fe-a14e-d3f32f0fd27d None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:13.207798 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9d417da-0f57-49fe-a14e-d3f32f0fd27d None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:13.214636 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c7873c8-b3ad-4589-8a28-04e7bfa8b44a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:13.215258 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c7873c8-b3ad-4589-8a28-04e7bfa8b44a admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:13.215534 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c7873c8-b3ad-4589-8a28-04e7bfa8b44a admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:13.215804 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c7873c8-b3ad-4589-8a28-04e7bfa8b44a admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:13.216478 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c7873c8-b3ad-4589-8a28-04e7bfa8b44a admin admin] RBAC: Adding query filter params (name=admin) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:13.216779 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c7873c8-b3ad-4589-8a28-04e7bfa8b44a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:13.218722 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c7873c8-b3ad-4589-8a28-04e7bfa8b44a admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:13.231066 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 143/293] 10.222.0.36 () {56 vars in 1120 bytes} [Thu Oct 20 08:48:13 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:13.236615 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebfc017a-8bbb-4f3a-a0eb-9c267b313501 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:13.247348 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebfc017a-8bbb-4f3a-a0eb-9c267b313501 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:13.254452 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d8a2ffb-9eeb-4ac4-850c-470134f11155 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:13.255114 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8a2ffb-9eeb-4ac4-850c-470134f11155 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:13.255428 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8a2ffb-9eeb-4ac4-850c-470134f11155 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:13.255698 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8a2ffb-9eeb-4ac4-850c-470134f11155 admin admin] PATH_INFO: `/v3/groups/0cef02e6800043a583f287e2bfc884b1` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:13.267104 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d8a2ffb-9eeb-4ac4-850c-470134f11155 admin admin] RBAC: Authorizing `identity:get_group(group_id=0cef02e6800043a583f287e2bfc884b1)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:13.269052 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d8a2ffb-9eeb-4ac4-850c-470134f11155 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:13.270740 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 151/294] 10.222.0.36 () {56 vars in 1201 bytes} [Thu Oct 20 08:48:13 2022] GET /identity/v3/groups/0cef02e6800043a583f287e2bfc884b1 => generated 229 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:13.276297 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c7873c8-b3ad-4589-8a28-04e7bfa8b44a admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:13.286386 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c7873c8-b3ad-4589-8a28-04e7bfa8b44a admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:13.294534 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c9d188d-84c1-4623-9f4d-a6812314916b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:13.295160 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c9d188d-84c1-4623-9f4d-a6812314916b admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:13.295439 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c9d188d-84c1-4623-9f4d-a6812314916b admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:13.295712 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c9d188d-84c1-4623-9f4d-a6812314916b admin admin] PATH_INFO: `/v3/projects/602743e1ebcd495bbe3a73b9f8abba4e` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:13.296855 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c9d188d-84c1-4623-9f4d-a6812314916b admin admin] RBAC: Authorizing `identity:get_project(project_id=602743e1ebcd495bbe3a73b9f8abba4e)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:13.298682 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c9d188d-84c1-4623-9f4d-a6812314916b admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:13.300483 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 144/295] 10.222.0.36 () {56 vars in 1207 bytes} [Thu Oct 20 08:48:13 2022] GET /identity/v3/projects/602743e1ebcd495bbe3a73b9f8abba4e => generated 344 bytes in 24 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:13.306554 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d8a2ffb-9eeb-4ac4-850c-470134f11155 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:13.316764 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d8a2ffb-9eeb-4ac4-850c-470134f11155 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:13.326546 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1276b8db-d54c-439a-b2a1-ceeb0b08be34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:13.327210 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1276b8db-d54c-439a-b2a1-ceeb0b08be34 admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:13.327488 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1276b8db-d54c-439a-b2a1-ceeb0b08be34 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:13.327756 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1276b8db-d54c-439a-b2a1-ceeb0b08be34 admin admin] PATH_INFO: `/v3/projects/602743e1ebcd495bbe3a73b9f8abba4e/groups/0cef02e6800043a583f287e2bfc884b1/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:13.329678 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1276b8db-d54c-439a-b2a1-ceeb0b08be34 admin admin] RBAC: Authorizing `identity:create_grant(project_id=602743e1ebcd495bbe3a73b9f8abba4e, group_id=0cef02e6800043a583f287e2bfc884b1, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:13.332388 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1276b8db-d54c-439a-b2a1-ceeb0b08be34 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:13.353844 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 152/296] 10.222.0.36 () {58 vars in 1463 bytes} [Thu Oct 20 08:48:13 2022] PUT /identity/v3/projects/602743e1ebcd495bbe3a73b9f8abba4e/groups/0cef02e6800043a583f287e2bfc884b1/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 08:48:14.093509 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64aa522b-26f3-4e15-94e8-fa2d21b5daa5 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:14.093927 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64aa522b-26f3-4e15-94e8-fa2d21b5daa5 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:14.094121 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64aa522b-26f3-4e15-94e8-fa2d21b5daa5 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:14.095425 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 145/297] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:48:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:48:14.102751 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a3ac24d-6c39-4e72-8a6e-0cbbe672b87f None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:14.103041 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a3ac24d-6c39-4e72-8a6e-0cbbe672b87f None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:14.103316 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a3ac24d-6c39-4e72-8a6e-0cbbe672b87f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:14.125233 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-7a3ac24d-6c39-4e72-8a6e-0cbbe672b87f None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:48:14.126417 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-7a3ac24d-6c39-4e72-8a6e-0cbbe672b87f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:48:14.200798 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:14.206396 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 153/298] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:48:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 20 08:48:14.334601 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4fb8007-4bd7-42a7-9543-1ea2e791d441 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:14.335254 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4fb8007-4bd7-42a7-9543-1ea2e791d441 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:14.335254 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4fb8007-4bd7-42a7-9543-1ea2e791d441 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:14.336490 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 146/299] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:48:14 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:48:14.342099 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6414179-672f-417f-83b3-e79755918ae2 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:14.342360 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6414179-672f-417f-83b3-e79755918ae2 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:14.342667 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6414179-672f-417f-83b3-e79755918ae2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:14.379525 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-f6414179-672f-417f-83b3-e79755918ae2 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:48:14.380837 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-f6414179-672f-417f-83b3-e79755918ae2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:48:14.468056 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 154/300] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:48:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 20 08:48:14.646369 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4fb8007-4bd7-42a7-9543-1ea2e791d441 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:14.691952 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:14.752256 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4fb8007-4bd7-42a7-9543-1ea2e791d441 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:14.792764 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af6a12df-ac78-4301-bffc-0520401c6f12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:14.793408 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6a12df-ac78-4301-bffc-0520401c6f12 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:14.793685 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6a12df-ac78-4301-bffc-0520401c6f12 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:14.794017 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6a12df-ac78-4301-bffc-0520401c6f12 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:14.806457 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af6a12df-ac78-4301-bffc-0520401c6f12 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 20 08:48:14.811777 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-af6a12df-ac78-4301-bffc-0520401c6f12 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 20 08:48:14.811777 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:14.811777 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:14.811777 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:14.811777 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:14.811777 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:14.811777 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:14.811777 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:14.811777 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:14.811777 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:14.811777 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:14.811777 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:14.811777 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 20 08:48:14.811777 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 20 08:48:14.811777 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 20 08:48:14.811777 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 20 08:48:14.812668 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 20 08:48:14.812668 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 20 08:48:14.812668 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 20 08:48:14.812668 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 20 08:48:14.812668 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:14.812668 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:14.812668 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:14.812668 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:14.812668 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:14.812668 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:14.812668 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:14.812668 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:14.812668 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:14.812668 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:14.812668 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:14.812668 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:14.813469 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:14.813469 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:14.813469 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:14.813469 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:14.813469 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 20 08:48:14.813469 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 20 08:48:14.813469 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 20 08:48:14.813469 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 20 08:48:14.813469 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 20 08:48:14.813469 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 20 08:48:14.813469 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 20 08:48:14.813469 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:14.813469 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 147/301] 10.222.0.36 () {56 vars in 1099 bytes} [Thu Oct 20 08:48:14 2022] GET /identity/v3/roles/member => generated 84 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:14.818741 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6414179-672f-417f-83b3-e79755918ae2 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:14.893222 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d12ab0c-48b7-46d4-bdde-214f79d1fb33 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:14.893539 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d12ab0c-48b7-46d4-bdde-214f79d1fb33 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:14.893764 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d12ab0c-48b7-46d4-bdde-214f79d1fb33 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:14.895125 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 148/302] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:48:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:48:14.902803 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f0ef7b-83df-4078-9918-7ed2a61ea3cf None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:14.903025 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f0ef7b-83df-4078-9918-7ed2a61ea3cf None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:14.903280 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f0ef7b-83df-4078-9918-7ed2a61ea3cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:14.934636 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-78f0ef7b-83df-4078-9918-7ed2a61ea3cf None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:48:14.935830 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-78f0ef7b-83df-4078-9918-7ed2a61ea3cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:48:14.963548 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6414179-672f-417f-83b3-e79755918ae2 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:14.999364 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:15.005632 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa509128-6dc4-4e24-96d5-f055f08095e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:15.006299 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa509128-6dc4-4e24-96d5-f055f08095e1 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:15.006798 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa509128-6dc4-4e24-96d5-f055f08095e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:15.007234 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa509128-6dc4-4e24-96d5-f055f08095e1 None admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:15.008115 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa509128-6dc4-4e24-96d5-f055f08095e1 None admin] RBAC: Adding query filter params (name=member) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:15.008585 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa509128-6dc4-4e24-96d5-f055f08095e1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:15.010124 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa509128-6dc4-4e24-96d5-f055f08095e1 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:15.019314 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 155/303] 10.222.0.36 () {56 vars in 1101 bytes} [Thu Oct 20 08:48:14 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 08:48:15.020366 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa509128-6dc4-4e24-96d5-f055f08095e1 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:15.044268 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 149/304] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:48:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 20 08:48:15.045285 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78f0ef7b-83df-4078-9918-7ed2a61ea3cf None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:15.085551 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:15.158086 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78f0ef7b-83df-4078-9918-7ed2a61ea3cf None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:15.197287 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deb03223-188f-4e99-a5d8-cfb53023c445 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:15.198004 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deb03223-188f-4e99-a5d8-cfb53023c445 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:15.198356 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deb03223-188f-4e99-a5d8-cfb53023c445 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:15.198726 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deb03223-188f-4e99-a5d8-cfb53023c445 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:15.251269 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa509128-6dc4-4e24-96d5-f055f08095e1 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:15.276177 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deb03223-188f-4e99-a5d8-cfb53023c445 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:15.277943 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deb03223-188f-4e99-a5d8-cfb53023c445 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:15.322887 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:15.330595 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c693f06-5f8c-42f5-a162-785b5a4d9d9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:15.331243 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c693f06-5f8c-42f5-a162-785b5a4d9d9e None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:15.331532 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c693f06-5f8c-42f5-a162-785b5a4d9d9e None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:15.331771 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c693f06-5f8c-42f5-a162-785b5a4d9d9e None admin] PATH_INFO: `/v3/roles/member` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:15.349180 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c693f06-5f8c-42f5-a162-785b5a4d9d9e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 20 08:48:15.359340 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-2c693f06-5f8c-42f5-a162-785b5a4d9d9e None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 20 08:48:15.359340 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:15.359340 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:15.359340 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:15.359340 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:15.359340 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:15.359340 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:15.359340 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:15.359340 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:15.359340 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:15.359340 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:15.359340 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:15.359340 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 20 08:48:15.359340 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 20 08:48:15.359340 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 20 08:48:15.359340 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 20 08:48:15.360333 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 20 08:48:15.360333 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 20 08:48:15.360333 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 20 08:48:15.360333 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 20 08:48:15.360333 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:15.360333 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:15.360333 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:15.360333 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:15.360333 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:15.360333 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:15.360333 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:15.360333 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:15.360333 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:15.360333 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:15.360333 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:15.360333 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:15.361161 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:15.361161 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:15.361161 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:15.361161 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:15.361161 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 20 08:48:15.361161 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 20 08:48:15.361161 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 20 08:48:15.361161 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 20 08:48:15.361161 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 20 08:48:15.361161 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 20 08:48:15.361161 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 20 08:48:15.361161 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:48:15.361161 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 156/305] 10.222.0.36 () {56 vars in 1099 bytes} [Thu Oct 20 08:48:15 2022] GET /identity/v3/roles/member => generated 84 bytes in 341 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:15.380323 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c693f06-5f8c-42f5-a162-785b5a4d9d9e None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:15.392807 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 150/306] 10.222.0.36 () {58 vars in 1282 bytes} [Thu Oct 20 08:48:15 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 348 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 20 08:48:15.398874 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deb03223-188f-4e99-a5d8-cfb53023c445 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:15.483836 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:15.512116 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deb03223-188f-4e99-a5d8-cfb53023c445 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:15.536268 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c693f06-5f8c-42f5-a162-785b5a4d9d9e None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:15.551260 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-137722dc-c4d9-4e5b-b34a-c13cbe3ef845 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:15.551888 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-137722dc-c4d9-4e5b-b34a-c13cbe3ef845 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:15.552167 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-137722dc-c4d9-4e5b-b34a-c13cbe3ef845 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:15.553055 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-137722dc-c4d9-4e5b-b34a-c13cbe3ef845 None admin] PATH_INFO: `/v3/projects/fcebd94752274a34a2068dd59f6fe700` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:15.554277 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-137722dc-c4d9-4e5b-b34a-c13cbe3ef845 None admin] RBAC: Authorizing `identity:get_project(project_id=fcebd94752274a34a2068dd59f6fe700)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:15.556137 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-137722dc-c4d9-4e5b-b34a-c13cbe3ef845 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:15.557968 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 151/307] 10.222.0.36 () {56 vars in 1186 bytes} [Thu Oct 20 08:48:15 2022] GET /identity/v3/projects/fcebd94752274a34a2068dd59f6fe700 => generated 298 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:15.559477 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bd6c08e-2217-4991-8569-520a13c98805 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:15.559477 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bd6c08e-2217-4991-8569-520a13c98805 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:15.559671 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bd6c08e-2217-4991-8569-520a13c98805 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:15.561598 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 152/308] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:48:15 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:48:15.561598 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bd6c08e-2217-4991-8569-520a13c98805 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:15.577878 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:15.579648 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c116741-9a08-4083-b63c-2b33ab017a68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:15.580245 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c116741-9a08-4083-b63c-2b33ab017a68 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:15.580509 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c116741-9a08-4083-b63c-2b33ab017a68 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:15.580804 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c116741-9a08-4083-b63c-2b33ab017a68 None admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:15.581435 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c116741-9a08-4083-b63c-2b33ab017a68 None admin] RBAC: Adding query filter params (name=member) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:15.581731 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c116741-9a08-4083-b63c-2b33ab017a68 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:15.584527 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c116741-9a08-4083-b63c-2b33ab017a68 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:15.593290 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 157/309] 10.222.0.36 () {56 vars in 1101 bytes} [Thu Oct 20 08:48:15 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:15.596190 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c116741-9a08-4083-b63c-2b33ab017a68 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:15.697096 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bd6c08e-2217-4991-8569-520a13c98805 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:15.729635 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c116741-9a08-4083-b63c-2b33ab017a68 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:15.745949 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99464382-331e-4f8b-b124-9f26a055d4a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:15.746571 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99464382-331e-4f8b-b124-9f26a055d4a1 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:15.746853 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99464382-331e-4f8b-b124-9f26a055d4a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:15.747124 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99464382-331e-4f8b-b124-9f26a055d4a1 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:15.748520 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:15.763738 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99464382-331e-4f8b-b124-9f26a055d4a1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 20 08:48:15.769923 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-99464382-331e-4f8b-b124-9f26a055d4a1 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 20 08:48:15.769923 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:15.769923 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:15.769923 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:15.769923 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:15.769923 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:15.769923 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:15.769923 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:15.769923 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:15.769923 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:15.769923 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:15.769923 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:15.769923 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 20 08:48:15.769923 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 20 08:48:15.769923 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 20 08:48:15.769923 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 20 08:48:15.771081 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 20 08:48:15.771081 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 20 08:48:15.771081 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 20 08:48:15.771081 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 20 08:48:15.771081 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:15.771081 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:15.771081 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:15.771081 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:15.771081 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:15.771081 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:15.771081 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:15.771081 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:15.771081 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:15.771081 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:15.771081 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:15.771081 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:15.771910 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:15.771910 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:15.771910 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:15.771910 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:15.771910 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 20 08:48:15.771910 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 20 08:48:15.771910 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 20 08:48:15.771910 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 20 08:48:15.771910 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 20 08:48:15.771910 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 20 08:48:15.771910 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 20 08:48:15.771910 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:15.771910 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 153/310] 10.222.0.36 () {56 vars in 1114 bytes} [Thu Oct 20 08:48:15 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:15.773145 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-469f62d2-21c5-4101-ab30-024e8de51087 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:15.773245 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-469f62d2-21c5-4101-ab30-024e8de51087 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:15.776749 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f116b0-de3c-41d2-ac7a-0632ea28e0e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:15.777380 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f116b0-de3c-41d2-ac7a-0632ea28e0e8 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:15.777653 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f116b0-de3c-41d2-ac7a-0632ea28e0e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:15.778127 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f116b0-de3c-41d2-ac7a-0632ea28e0e8 None admin] PATH_INFO: `/v3/groups/96d598164a48402eb014d5f9cbad95f8` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:15.779456 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-469f62d2-21c5-4101-ab30-024e8de51087 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:15.779564 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52f116b0-de3c-41d2-ac7a-0632ea28e0e8 None admin] RBAC: Authorizing `identity:get_group(group_id=96d598164a48402eb014d5f9cbad95f8)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:15.780536 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52f116b0-de3c-41d2-ac7a-0632ea28e0e8 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:15.782310 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 158/311] 10.222.0.36 () {56 vars in 1180 bytes} [Thu Oct 20 08:48:15 2022] GET /identity/v3/groups/96d598164a48402eb014d5f9cbad95f8 => generated 226 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:15.784667 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f116b0-de3c-41d2-ac7a-0632ea28e0e8 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:15.801708 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-469f62d2-21c5-4101-ab30-024e8de51087 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:48:15.803510 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-469f62d2-21c5-4101-ab30-024e8de51087 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:48:15.882569 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:15.900319 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 154/312] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:48:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 20 08:48:15.901249 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-469f62d2-21c5-4101-ab30-024e8de51087 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:15.916595 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f116b0-de3c-41d2-ac7a-0632ea28e0e8 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:15.954842 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b72f3a88-9573-4324-a32f-a08fadebb981 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:15.955473 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b72f3a88-9573-4324-a32f-a08fadebb981 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:15.956108 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b72f3a88-9573-4324-a32f-a08fadebb981 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:15.956379 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b72f3a88-9573-4324-a32f-a08fadebb981 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:16.017559 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-469f62d2-21c5-4101-ab30-024e8de51087 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:16.023271 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:16.042700 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b72f3a88-9573-4324-a32f-a08fadebb981 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:16.044186 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b72f3a88-9573-4324-a32f-a08fadebb981 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:16.055415 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aad336f-8a6f-4dc0-b49e-4b7413cb9e76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:16.056023 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aad336f-8a6f-4dc0-b49e-4b7413cb9e76 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:16.056298 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aad336f-8a6f-4dc0-b49e-4b7413cb9e76 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:16.056563 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aad336f-8a6f-4dc0-b49e-4b7413cb9e76 None admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:16.057223 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aad336f-8a6f-4dc0-b49e-4b7413cb9e76 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:16.057518 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aad336f-8a6f-4dc0-b49e-4b7413cb9e76 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:16.058929 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aad336f-8a6f-4dc0-b49e-4b7413cb9e76 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:16.069759 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 155/313] 10.222.0.36 () {56 vars in 1111 bytes} [Thu Oct 20 08:48:15 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:16.070696 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aad336f-8a6f-4dc0-b49e-4b7413cb9e76 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:16.134982 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:16.157717 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 159/314] 10.222.0.36 () {58 vars in 1282 bytes} [Thu Oct 20 08:48:15 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 375 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 20 08:48:16.158648 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b72f3a88-9573-4324-a32f-a08fadebb981 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:16.192000 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aad336f-8a6f-4dc0-b49e-4b7413cb9e76 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:16.229791 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0506a336-baf8-4e8b-a80f-b2747b752d6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:16.230393 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0506a336-baf8-4e8b-a80f-b2747b752d6e None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:16.230767 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0506a336-baf8-4e8b-a80f-b2747b752d6e None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:16.231052 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0506a336-baf8-4e8b-a80f-b2747b752d6e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:16.231760 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0506a336-baf8-4e8b-a80f-b2747b752d6e None admin] RBAC: Adding query filter params (group.id=96d598164a48402eb014d5f9cbad95f8, role.id=747fbb4dfbec43baa31ec3b7761fd4b1, scope.project.id=fcebd94752274a34a2068dd59f6fe700) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:16.232111 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0506a336-baf8-4e8b-a80f-b2747b752d6e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:16.233595 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0506a336-baf8-4e8b-a80f-b2747b752d6e None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:16.239686 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 156/315] 10.222.0.36 () {56 vars in 1376 bytes} [Thu Oct 20 08:48:16 2022] GET /identity/v3/role_assignments?group.id=96d598164a48402eb014d5f9cbad95f8&scope.project.id=fcebd94752274a34a2068dd59f6fe700&role.id=747fbb4dfbec43baa31ec3b7761fd4b1 => generated 611 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:16.242257 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0506a336-baf8-4e8b-a80f-b2747b752d6e None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:16.257959 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:16.280318 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b72f3a88-9573-4324-a32f-a08fadebb981 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:16.325924 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-557ab717-a6c1-4e4b-988d-ab000f1d1b9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:16.326596 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-557ab717-a6c1-4e4b-988d-ab000f1d1b9d None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:16.326877 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-557ab717-a6c1-4e4b-988d-ab000f1d1b9d None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:16.327150 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-557ab717-a6c1-4e4b-988d-ab000f1d1b9d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:16.388649 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0506a336-baf8-4e8b-a80f-b2747b752d6e None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:16.413653 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:16.423838 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-557ab717-a6c1-4e4b-988d-ab000f1d1b9d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:16.425354 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-557ab717-a6c1-4e4b-988d-ab000f1d1b9d None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:16.441281 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d067f8ff-6586-4db8-b03e-9d2adaa7d46d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:16.441922 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d067f8ff-6586-4db8-b03e-9d2adaa7d46d None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:16.442200 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d067f8ff-6586-4db8-b03e-9d2adaa7d46d None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:16.442662 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d067f8ff-6586-4db8-b03e-9d2adaa7d46d None admin] PATH_INFO: `/v3/projects/a00517b9cca94187ac4aba4ec7468007` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:16.444532 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d067f8ff-6586-4db8-b03e-9d2adaa7d46d None admin] RBAC: Authorizing `identity:get_project(project_id=a00517b9cca94187ac4aba4ec7468007)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:16.445688 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d067f8ff-6586-4db8-b03e-9d2adaa7d46d None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:16.448514 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 157/316] 10.222.0.36 () {56 vars in 1186 bytes} [Thu Oct 20 08:48:16 2022] GET /identity/v3/projects/a00517b9cca94187ac4aba4ec7468007 => generated 302 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:16.448514 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d067f8ff-6586-4db8-b03e-9d2adaa7d46d None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:16.524094 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:16.544614 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 160/317] 10.222.0.36 () {58 vars in 1282 bytes} [Thu Oct 20 08:48:16 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 386 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 20 08:48:16.546296 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed477c5-2532-4ef9-942e-fe881a973758 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:16.546622 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed477c5-2532-4ef9-942e-fe881a973758 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:16.546894 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed477c5-2532-4ef9-942e-fe881a973758 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:16.548262 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 161/318] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:48:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:48:16.548877 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ed477c5-2532-4ef9-942e-fe881a973758 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:16.577276 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d067f8ff-6586-4db8-b03e-9d2adaa7d46d None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:16.617571 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be570b7e-e4d4-459d-a871-d218ad7c8018 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:16.618207 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be570b7e-e4d4-459d-a871-d218ad7c8018 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:16.618504 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be570b7e-e4d4-459d-a871-d218ad7c8018 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:16.618831 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be570b7e-e4d4-459d-a871-d218ad7c8018 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:16.631057 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be570b7e-e4d4-459d-a871-d218ad7c8018 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 20 08:48:16.637328 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-be570b7e-e4d4-459d-a871-d218ad7c8018 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 20 08:48:16.637328 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:16.637328 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:16.637328 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:16.637328 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:16.637328 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:16.637328 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:16.637328 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:16.637328 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:16.637328 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:16.637328 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:16.637328 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:16.637328 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 20 08:48:16.637328 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 20 08:48:16.637328 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 20 08:48:16.637328 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 20 08:48:16.638213 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 20 08:48:16.638213 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 20 08:48:16.638213 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 20 08:48:16.638213 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 20 08:48:16.638213 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:16.638213 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:16.638213 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:16.638213 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:16.638213 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:16.638213 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:16.638213 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:16.638213 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:16.638213 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:16.638213 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:16.638213 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:16.638213 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:16.639057 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:16.639057 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:16.639057 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:16.639057 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:16.639057 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 20 08:48:16.639057 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 20 08:48:16.639057 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 20 08:48:16.639057 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 20 08:48:16.639057 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 20 08:48:16.639057 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 20 08:48:16.639057 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 20 08:48:16.639057 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:16.639057 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 158/319] 10.222.0.36 () {56 vars in 1114 bytes} [Thu Oct 20 08:48:16 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:16.639057 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be570b7e-e4d4-459d-a871-d218ad7c8018 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:16.672151 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ed477c5-2532-4ef9-942e-fe881a973758 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:16.690789 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:16.716988 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-533daab3-0bb2-4111-b629-cf7ea9dac4ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:16.717615 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533daab3-0bb2-4111-b629-cf7ea9dac4ff None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:16.717882 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533daab3-0bb2-4111-b629-cf7ea9dac4ff None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:16.718144 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533daab3-0bb2-4111-b629-cf7ea9dac4ff None admin] PATH_INFO: `/v3/groups/96d598164a48402eb014d5f9cbad95f8` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:16.719290 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-533daab3-0bb2-4111-b629-cf7ea9dac4ff None admin] RBAC: Authorizing `identity:get_group(group_id=96d598164a48402eb014d5f9cbad95f8)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:16.720794 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-533daab3-0bb2-4111-b629-cf7ea9dac4ff None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:16.722955 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 162/320] 10.222.0.36 () {56 vars in 1180 bytes} [Thu Oct 20 08:48:16 2022] GET /identity/v3/groups/96d598164a48402eb014d5f9cbad95f8 => generated 226 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:16.725260 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f901aa-3519-4efe-bcc6-7f866bcbd0da None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:16.726327 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f901aa-3519-4efe-bcc6-7f866bcbd0da None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:16.726327 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f901aa-3519-4efe-bcc6-7f866bcbd0da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:16.744715 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-f1f901aa-3519-4efe-bcc6-7f866bcbd0da None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:48:16.745903 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-f1f901aa-3519-4efe-bcc6-7f866bcbd0da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:48:16.758175 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be570b7e-e4d4-459d-a871-d218ad7c8018 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:16.796467 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd9a3f76-51c6-4cc1-9727-fdeba6a0ac9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:16.797023 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd9a3f76-51c6-4cc1-9727-fdeba6a0ac9c None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:16.797301 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd9a3f76-51c6-4cc1-9727-fdeba6a0ac9c None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:16.797570 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd9a3f76-51c6-4cc1-9727-fdeba6a0ac9c None admin] PATH_INFO: `/v3/projects/fcebd94752274a34a2068dd59f6fe700` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:16.798725 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd9a3f76-51c6-4cc1-9727-fdeba6a0ac9c None admin] RBAC: Authorizing `identity:get_project(project_id=fcebd94752274a34a2068dd59f6fe700)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:16.800176 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd9a3f76-51c6-4cc1-9727-fdeba6a0ac9c None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:16.801795 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 159/321] 10.222.0.36 () {56 vars in 1186 bytes} [Thu Oct 20 08:48:16 2022] GET /identity/v3/projects/fcebd94752274a34a2068dd59f6fe700 => generated 298 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:16.803221 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd9a3f76-51c6-4cc1-9727-fdeba6a0ac9c None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:16.805378 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:16.830769 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 163/322] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:48:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 20 08:48:16.831942 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f901aa-3519-4efe-bcc6-7f866bcbd0da None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:16.923613 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd9a3f76-51c6-4cc1-9727-fdeba6a0ac9c None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:16.941917 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:16.968411 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f901aa-3519-4efe-bcc6-7f866bcbd0da None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:16.971331 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-814be5f0-c3a6-4378-a749-2a50388d1a88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:16.971824 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-814be5f0-c3a6-4378-a749-2a50388d1a88 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:16.972111 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-814be5f0-c3a6-4378-a749-2a50388d1a88 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:16.972386 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-814be5f0-c3a6-4378-a749-2a50388d1a88 None admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:16.973060 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-814be5f0-c3a6-4378-a749-2a50388d1a88 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:16.973363 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-814be5f0-c3a6-4378-a749-2a50388d1a88 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:16.976125 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-814be5f0-c3a6-4378-a749-2a50388d1a88 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:16.984656 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 160/323] 10.222.0.36 () {56 vars in 1111 bytes} [Thu Oct 20 08:48:16 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:16.987392 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-814be5f0-c3a6-4378-a749-2a50388d1a88 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:17.010262 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ef7dddb-ec93-435b-8166-309c4f2e038e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:17.010908 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ef7dddb-ec93-435b-8166-309c4f2e038e None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:17.011190 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ef7dddb-ec93-435b-8166-309c4f2e038e None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:17.011461 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ef7dddb-ec93-435b-8166-309c4f2e038e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:17.012637 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ef7dddb-ec93-435b-8166-309c4f2e038e None admin] RBAC: Adding query filter params (group.id=96d598164a48402eb014d5f9cbad95f8, role.id=747fbb4dfbec43baa31ec3b7761fd4b1, scope.project.id=a00517b9cca94187ac4aba4ec7468007) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:17.013234 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ef7dddb-ec93-435b-8166-309c4f2e038e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:17.017980 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ef7dddb-ec93-435b-8166-309c4f2e038e None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:17.023590 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 164/324] 10.222.0.36 () {56 vars in 1376 bytes} [Thu Oct 20 08:48:16 2022] GET /identity/v3/role_assignments?group.id=96d598164a48402eb014d5f9cbad95f8&scope.project.id=a00517b9cca94187ac4aba4ec7468007&role.id=747fbb4dfbec43baa31ec3b7761fd4b1 => generated 611 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:17.024422 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ef7dddb-ec93-435b-8166-309c4f2e038e None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:17.101844 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:17.138889 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-814be5f0-c3a6-4378-a749-2a50388d1a88 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:17.167598 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ef7dddb-ec93-435b-8166-309c4f2e038e None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:17.176518 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bc22b17-93d9-4ea1-adf3-28711847927e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:17.177169 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc22b17-93d9-4ea1-adf3-28711847927e None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:17.178017 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc22b17-93d9-4ea1-adf3-28711847927e None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:17.178720 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc22b17-93d9-4ea1-adf3-28711847927e None admin] PATH_INFO: `/v3/groups/96d598164a48402eb014d5f9cbad95f8` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:17.180250 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bc22b17-93d9-4ea1-adf3-28711847927e None admin] RBAC: Authorizing `identity:get_group(group_id=96d598164a48402eb014d5f9cbad95f8)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:17.181759 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bc22b17-93d9-4ea1-adf3-28711847927e None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:17.189560 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 161/325] 10.222.0.36 () {56 vars in 1180 bytes} [Thu Oct 20 08:48:16 2022] GET /identity/v3/groups/96d598164a48402eb014d5f9cbad95f8 => generated 226 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:17.189560 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bc22b17-93d9-4ea1-adf3-28711847927e None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:17.209247 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d0950da-100a-407b-8754-b59a7ccc5340 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:17.209856 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d0950da-100a-407b-8754-b59a7ccc5340 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:17.210128 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d0950da-100a-407b-8754-b59a7ccc5340 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:17.210388 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d0950da-100a-407b-8754-b59a7ccc5340 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:17.241724 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:17.292067 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d0950da-100a-407b-8754-b59a7ccc5340 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:17.293541 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d0950da-100a-407b-8754-b59a7ccc5340 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:17.303988 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bc22b17-93d9-4ea1-adf3-28711847927e None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:17.336602 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc04329e-06a1-4f89-8bab-cd8f15513bcc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:17.336796 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:17.337217 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc04329e-06a1-4f89-8bab-cd8f15513bcc None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:17.337492 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc04329e-06a1-4f89-8bab-cd8f15513bcc None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:17.337758 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc04329e-06a1-4f89-8bab-cd8f15513bcc None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:17.347767 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc04329e-06a1-4f89-8bab-cd8f15513bcc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 20 08:48:17.353065 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-dc04329e-06a1-4f89-8bab-cd8f15513bcc None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 20 08:48:17.353065 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:17.353065 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:17.353065 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:17.353065 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:17.353065 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:17.353065 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:17.353065 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:17.353065 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:17.353065 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:17.353065 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:17.353065 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:17.353065 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 20 08:48:17.353065 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 20 08:48:17.353065 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 20 08:48:17.353065 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 20 08:48:17.354001 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 20 08:48:17.354001 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 20 08:48:17.354001 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 20 08:48:17.354001 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 20 08:48:17.354001 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:17.354001 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:17.354001 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:17.354001 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:17.354001 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:17.354001 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:17.354001 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:17.354001 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:17.354001 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:17.354001 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:17.354001 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:17.354001 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:17.355123 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:17.355123 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:17.355123 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:17.355123 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:17.355123 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 20 08:48:17.355123 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 20 08:48:17.355123 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 20 08:48:17.355123 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 20 08:48:17.355123 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 20 08:48:17.355123 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 20 08:48:17.355123 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 20 08:48:17.355123 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:17.355123 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 162/326] 10.222.0.36 () {56 vars in 1096 bytes} [Thu Oct 20 08:48:17 2022] GET /identity/v3/roles/admin => generated 83 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:17.355123 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc04329e-06a1-4f89-8bab-cd8f15513bcc None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:17.394709 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 165/327] 10.222.0.36 () {58 vars in 1282 bytes} [Thu Oct 20 08:48:17 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 20 08:48:17.395555 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d0950da-100a-407b-8754-b59a7ccc5340 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:17.457568 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc04329e-06a1-4f89-8bab-cd8f15513bcc None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:17.460141 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:17.491582 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f38c461-deb8-4b53-afce-f10ed88e0a60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:17.491903 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d0950da-100a-407b-8754-b59a7ccc5340 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:17.492172 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f38c461-deb8-4b53-afce-f10ed88e0a60 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:17.492447 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f38c461-deb8-4b53-afce-f10ed88e0a60 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:17.492713 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f38c461-deb8-4b53-afce-f10ed88e0a60 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:17.493426 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f38c461-deb8-4b53-afce-f10ed88e0a60 None admin] RBAC: Adding query filter params (group.id=96d598164a48402eb014d5f9cbad95f8, role.id=b3f8d63bf4c4488288e57693c6e9804c, scope.project.id=fcebd94752274a34a2068dd59f6fe700) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:17.493725 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f38c461-deb8-4b53-afce-f10ed88e0a60 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:17.495175 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f38c461-deb8-4b53-afce-f10ed88e0a60 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:17.500715 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 163/328] 10.222.0.36 () {56 vars in 1376 bytes} [Thu Oct 20 08:48:17 2022] GET /identity/v3/role_assignments?group.id=96d598164a48402eb014d5f9cbad95f8&scope.project.id=fcebd94752274a34a2068dd59f6fe700&role.id=b3f8d63bf4c4488288e57693c6e9804c => generated 611 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:17.501562 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f38c461-deb8-4b53-afce-f10ed88e0a60 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:17.538837 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b69685b-2e6d-4098-a5fc-1b7d31b643d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:17.539525 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b69685b-2e6d-4098-a5fc-1b7d31b643d3 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:17.539800 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b69685b-2e6d-4098-a5fc-1b7d31b643d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:17.540064 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b69685b-2e6d-4098-a5fc-1b7d31b643d3 None admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:17.540739 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b69685b-2e6d-4098-a5fc-1b7d31b643d3 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:17.541035 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b69685b-2e6d-4098-a5fc-1b7d31b643d3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:17.542509 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b69685b-2e6d-4098-a5fc-1b7d31b643d3 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:17.551339 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 166/329] 10.222.0.36 () {56 vars in 1099 bytes} [Thu Oct 20 08:48:17 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:17.557660 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b69685b-2e6d-4098-a5fc-1b7d31b643d3 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:17.569276 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:17.615664 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f38c461-deb8-4b53-afce-f10ed88e0a60 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:17.653652 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-573cbd0e-f726-4110-8253-c55c15951410 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:17.654294 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-573cbd0e-f726-4110-8253-c55c15951410 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:17.654708 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-573cbd0e-f726-4110-8253-c55c15951410 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:17.654995 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-573cbd0e-f726-4110-8253-c55c15951410 None admin] PATH_INFO: `/v3/projects/a00517b9cca94187ac4aba4ec7468007` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:17.656142 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-573cbd0e-f726-4110-8253-c55c15951410 None admin] RBAC: Authorizing `identity:get_project(project_id=a00517b9cca94187ac4aba4ec7468007)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:17.657622 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-573cbd0e-f726-4110-8253-c55c15951410 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:17.663839 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 164/330] 10.222.0.36 () {56 vars in 1186 bytes} [Thu Oct 20 08:48:17 2022] GET /identity/v3/projects/a00517b9cca94187ac4aba4ec7468007 => generated 302 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:17.664918 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-573cbd0e-f726-4110-8253-c55c15951410 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:17.680254 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b69685b-2e6d-4098-a5fc-1b7d31b643d3 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:17.711679 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:17.722037 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a2de559-93ef-41c8-9424-7ca7570d4339 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:17.724949 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2de559-93ef-41c8-9424-7ca7570d4339 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:17.724949 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2de559-93ef-41c8-9424-7ca7570d4339 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:17.724949 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2de559-93ef-41c8-9424-7ca7570d4339 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:17.786091 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-573cbd0e-f726-4110-8253-c55c15951410 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:17.813269 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a2de559-93ef-41c8-9424-7ca7570d4339 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:17.813269 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a2de559-93ef-41c8-9424-7ca7570d4339 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:17.822027 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:17.827259 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d31053dc-3209-4122-83a8-9d3b8bd20e0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:17.828235 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d31053dc-3209-4122-83a8-9d3b8bd20e0e None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:17.828761 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d31053dc-3209-4122-83a8-9d3b8bd20e0e None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:17.829107 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d31053dc-3209-4122-83a8-9d3b8bd20e0e None admin] PATH_INFO: `/v3/groups/96d598164a48402eb014d5f9cbad95f8` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:17.830299 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d31053dc-3209-4122-83a8-9d3b8bd20e0e None admin] RBAC: Authorizing `identity:get_group(group_id=96d598164a48402eb014d5f9cbad95f8)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:17.831854 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d31053dc-3209-4122-83a8-9d3b8bd20e0e None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:17.833919 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 165/331] 10.222.0.36 () {56 vars in 1180 bytes} [Thu Oct 20 08:48:17 2022] GET /identity/v3/groups/96d598164a48402eb014d5f9cbad95f8 => generated 226 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:17.841301 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d31053dc-3209-4122-83a8-9d3b8bd20e0e None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:17.925919 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 167/332] 10.222.0.36 () {58 vars in 1282 bytes} [Thu Oct 20 08:48:17 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 368 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 20 08:48:17.931223 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a2de559-93ef-41c8-9424-7ca7570d4339 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:17.968804 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d31053dc-3209-4122-83a8-9d3b8bd20e0e None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:17.972311 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:18.011948 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27c0acb2-5f44-44a2-a815-01659e907967 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:18.012556 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c0acb2-5f44-44a2-a815-01659e907967 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:18.012829 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c0acb2-5f44-44a2-a815-01659e907967 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:18.013098 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c0acb2-5f44-44a2-a815-01659e907967 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:18.013813 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27c0acb2-5f44-44a2-a815-01659e907967 None admin] RBAC: Adding query filter params (group.id=96d598164a48402eb014d5f9cbad95f8, role.id=b3f8d63bf4c4488288e57693c6e9804c, scope.project.id=a00517b9cca94187ac4aba4ec7468007) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:18.014121 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27c0acb2-5f44-44a2-a815-01659e907967 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:18.015597 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27c0acb2-5f44-44a2-a815-01659e907967 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:18.026514 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 166/333] 10.222.0.36 () {56 vars in 1376 bytes} [Thu Oct 20 08:48:17 2022] GET /identity/v3/role_assignments?group.id=96d598164a48402eb014d5f9cbad95f8&scope.project.id=a00517b9cca94187ac4aba4ec7468007&role.id=b3f8d63bf4c4488288e57693c6e9804c => generated 611 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 08:48:18.043624 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a2de559-93ef-41c8-9424-7ca7570d4339 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:18.053615 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:18.084173 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73badd02-1609-4f2c-8479-08f9cdf905ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:18.084822 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73badd02-1609-4f2c-8479-08f9cdf905ab None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:18.085096 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73badd02-1609-4f2c-8479-08f9cdf905ab None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:18.085392 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73badd02-1609-4f2c-8479-08f9cdf905ab None admin] PATH_INFO: `/v3/projects/602743e1ebcd495bbe3a73b9f8abba4e` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:18.086579 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73badd02-1609-4f2c-8479-08f9cdf905ab None admin] RBAC: Authorizing `identity:get_project(project_id=602743e1ebcd495bbe3a73b9f8abba4e)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:18.088056 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73badd02-1609-4f2c-8479-08f9cdf905ab None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:18.089741 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 168/334] 10.222.0.36 () {56 vars in 1186 bytes} [Thu Oct 20 08:48:17 2022] GET /identity/v3/projects/602743e1ebcd495bbe3a73b9f8abba4e => generated 344 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 08:48:18.095499 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27c0acb2-5f44-44a2-a815-01659e907967 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:18.198334 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27c0acb2-5f44-44a2-a815-01659e907967 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:18.232458 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c56e8dec-c78b-4a17-905c-f1e724351344 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:18.233081 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c56e8dec-c78b-4a17-905c-f1e724351344 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:18.233361 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c56e8dec-c78b-4a17-905c-f1e724351344 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:18.233627 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c56e8dec-c78b-4a17-905c-f1e724351344 None admin] PATH_INFO: `/v3/groups/0cef02e6800043a583f287e2bfc884b1` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:18.234741 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c56e8dec-c78b-4a17-905c-f1e724351344 None admin] RBAC: Authorizing `identity:get_group(group_id=0cef02e6800043a583f287e2bfc884b1)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:18.236170 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c56e8dec-c78b-4a17-905c-f1e724351344 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:18.237753 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 167/335] 10.222.0.36 () {56 vars in 1180 bytes} [Thu Oct 20 08:48:18 2022] GET /identity/v3/groups/0cef02e6800043a583f287e2bfc884b1 => generated 229 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:18.243703 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73badd02-1609-4f2c-8479-08f9cdf905ab None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:18.342631 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73badd02-1609-4f2c-8479-08f9cdf905ab None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:18.381994 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-254d2c90-6762-4cd9-94f5-5fbe5755a98f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:18.382640 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254d2c90-6762-4cd9-94f5-5fbe5755a98f None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:18.382913 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254d2c90-6762-4cd9-94f5-5fbe5755a98f None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:18.383183 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254d2c90-6762-4cd9-94f5-5fbe5755a98f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:18.383914 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-254d2c90-6762-4cd9-94f5-5fbe5755a98f None admin] RBAC: Adding query filter params (group.id=0cef02e6800043a583f287e2bfc884b1, role.id=d9d3bc62f6e8453488c74856ef333bb3, scope.project.id=602743e1ebcd495bbe3a73b9f8abba4e) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:18.384224 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-254d2c90-6762-4cd9-94f5-5fbe5755a98f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:18.385659 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-254d2c90-6762-4cd9-94f5-5fbe5755a98f None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:18.391786 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 169/336] 10.222.0.36 () {56 vars in 1376 bytes} [Thu Oct 20 08:48:18 2022] GET /identity/v3/role_assignments?group.id=0cef02e6800043a583f287e2bfc884b1&scope.project.id=602743e1ebcd495bbe3a73b9f8abba4e&role.id=d9d3bc62f6e8453488c74856ef333bb3 => generated 611 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:21.022406 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d5483ec-32d9-444d-bdc6-b970d6507900 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:21.022938 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d5483ec-32d9-444d-bdc6-b970d6507900 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:21.022938 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d5483ec-32d9-444d-bdc6-b970d6507900 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:21.024534 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 168/337] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:48:21 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:48:21.029966 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba72219-d394-482f-acb3-adbc1ab1a193 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:21.030374 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba72219-d394-482f-acb3-adbc1ab1a193 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:21.030569 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba72219-d394-482f-acb3-adbc1ab1a193 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:21.055488 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-0ba72219-d394-482f-acb3-adbc1ab1a193 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:48:21.056766 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-0ba72219-d394-482f-acb3-adbc1ab1a193 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:48:21.107516 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:21.158274 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 170/338] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:48:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 20 08:48:21.329971 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7749e6-9a9e-4cab-b8f1-d18cf026f13f None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:21.330148 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7749e6-9a9e-4cab-b8f1-d18cf026f13f None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:21.330418 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7749e6-9a9e-4cab-b8f1-d18cf026f13f None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:21.332323 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 169/339] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:48:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:48:21.338281 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b754e5e-c0eb-434a-96b4-5eff46db5ef6 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:21.338731 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b754e5e-c0eb-434a-96b4-5eff46db5ef6 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:21.339001 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b754e5e-c0eb-434a-96b4-5eff46db5ef6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:21.369009 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-5b754e5e-c0eb-434a-96b4-5eff46db5ef6 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:48:21.370189 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-5b754e5e-c0eb-434a-96b4-5eff46db5ef6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:48:21.459598 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17aecc0-d918-4055-9e7c-b61ebf3b83cc None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:21.459943 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17aecc0-d918-4055-9e7c-b61ebf3b83cc None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:21.460150 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17aecc0-d918-4055-9e7c-b61ebf3b83cc None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:21.460783 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 171/340] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:48:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 20 08:48:21.461600 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 170/341] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:48:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:48:21.466979 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b0ab0d-6883-4c0b-aefe-c8a7c6542092 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:21.467263 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b0ab0d-6883-4c0b-aefe-c8a7c6542092 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:21.467535 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b0ab0d-6883-4c0b-aefe-c8a7c6542092 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:21.495777 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-d5b0ab0d-6883-4c0b-aefe-c8a7c6542092 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:48:21.496949 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-d5b0ab0d-6883-4c0b-aefe-c8a7c6542092 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:48:21.528259 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d17aecc0-d918-4055-9e7c-b61ebf3b83cc None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:21.588079 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:21.594864 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 172/342] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:48:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 20 08:48:21.676471 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d17aecc0-d918-4055-9e7c-b61ebf3b83cc None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:21.726439 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce255f79-32ec-443f-a461-531524e14891 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:21.727430 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce255f79-32ec-443f-a461-531524e14891 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:21.727909 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce255f79-32ec-443f-a461-531524e14891 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:21.728464 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce255f79-32ec-443f-a461-531524e14891 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:21.729907 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225a4b92-6ebb-4c66-9ff9-3ea5e11eabc7 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:21.730425 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225a4b92-6ebb-4c66-9ff9-3ea5e11eabc7 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:21.731037 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225a4b92-6ebb-4c66-9ff9-3ea5e11eabc7 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:21.732794 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 173/343] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:48:21 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:48:21.738120 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c461f5a-d2ff-4e45-b2b5-c0e0d49a7adb None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:21.738658 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c461f5a-d2ff-4e45-b2b5-c0e0d49a7adb None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:21.739151 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c461f5a-d2ff-4e45-b2b5-c0e0d49a7adb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:21.740762 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce255f79-32ec-443f-a461-531524e14891 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:21.742209 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce255f79-32ec-443f-a461-531524e14891 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:21.757829 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-ce255f79-32ec-443f-a461-531524e14891 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:21.757829 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:21.757829 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:21.757829 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:21.757829 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:21.757829 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:21.757829 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:21.757829 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:21.757829 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:21.757829 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:21.757829 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:21.757829 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:21.757829 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 20 08:48:21.757829 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 20 08:48:21.757829 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 20 08:48:21.757829 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 20 08:48:21.759048 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:21.759048 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:21.759048 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:21.759048 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:21.759048 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:21.759048 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:21.759048 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:21.759048 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:21.759048 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:21.759048 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:21.759048 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:21.759048 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:21.759048 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:21.759048 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:21.759048 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:21.759048 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:21.759048 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 20 08:48:21.760101 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 20 08:48:21.760101 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:21.760101 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:21.760101 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 171/344] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:48:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 20 08:48:21.766501 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce255f79-32ec-443f-a461-531524e14891 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:21.778169 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-0c461f5a-d2ff-4e45-b2b5-c0e0d49a7adb None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:48:21.779623 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-0c461f5a-d2ff-4e45-b2b5-c0e0d49a7adb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:48:21.821546 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:21.862830 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:21.875178 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce255f79-32ec-443f-a461-531524e14891 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:21.876159 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 174/345] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:48:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 20 08:48:21.877504 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c461f5a-d2ff-4e45-b2b5-c0e0d49a7adb None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:21.946947 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a65713e6-fc3b-403d-8904-4a3164e9387a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:21.947560 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a65713e6-fc3b-403d-8904-4a3164e9387a None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:21.947839 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a65713e6-fc3b-403d-8904-4a3164e9387a None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:21.948109 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a65713e6-fc3b-403d-8904-4a3164e9387a None admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:21.948749 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a65713e6-fc3b-403d-8904-4a3164e9387a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:21.949050 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a65713e6-fc3b-403d-8904-4a3164e9387a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:21.950419 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a65713e6-fc3b-403d-8904-4a3164e9387a None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:21.962703 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 172/346] 10.222.0.36 () {56 vars in 1109 bytes} [Thu Oct 20 08:48:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 08:48:21.963511 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a65713e6-fc3b-403d-8904-4a3164e9387a None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:22.012600 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c461f5a-d2ff-4e45-b2b5-c0e0d49a7adb None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:22.048552 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-721579d5-f982-493f-a299-76abb984007c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:22.049172 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-721579d5-f982-493f-a299-76abb984007c None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:22.049433 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-721579d5-f982-493f-a299-76abb984007c None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:22.049970 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-721579d5-f982-493f-a299-76abb984007c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:22.061332 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-721579d5-f982-493f-a299-76abb984007c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:22.063054 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-721579d5-f982-493f-a299-76abb984007c None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:22.071080 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a65713e6-fc3b-403d-8904-4a3164e9387a None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:22.080952 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-721579d5-f982-493f-a299-76abb984007c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:22.080952 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:22.080952 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:22.080952 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:22.080952 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:22.080952 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:22.080952 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:22.080952 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:22.080952 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:22.080952 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:22.080952 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:22.080952 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:22.080952 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 20 08:48:22.080952 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 20 08:48:22.080952 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 20 08:48:22.080952 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 20 08:48:22.081904 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:22.081904 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:22.081904 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:22.081904 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:22.081904 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:22.081904 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:22.081904 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:22.081904 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:22.081904 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:22.081904 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:22.081904 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:22.081904 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:22.081904 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:22.081904 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:22.081904 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:22.081904 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:22.081904 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 20 08:48:22.083076 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 20 08:48:22.083076 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:22.083076 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:48:22.085351 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 175/347] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:48:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:22.086400 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-721579d5-f982-493f-a299-76abb984007c None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:22.100158 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:22.106972 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93a3e291-836b-4137-aefc-9190cd73bd5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:22.107686 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a3e291-836b-4137-aefc-9190cd73bd5f None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:22.108042 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a3e291-836b-4137-aefc-9190cd73bd5f None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:22.108387 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a3e291-836b-4137-aefc-9190cd73bd5f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:22.120151 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93a3e291-836b-4137-aefc-9190cd73bd5f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:22.121633 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93a3e291-836b-4137-aefc-9190cd73bd5f None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:22.129836 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:22.137172 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-93a3e291-836b-4137-aefc-9190cd73bd5f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:22.137172 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:22.137172 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:22.137172 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:22.137172 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:22.137172 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:22.137172 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:22.137172 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:22.137172 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:22.137172 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:22.137172 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:22.137172 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:22.137172 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 20 08:48:22.137172 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 20 08:48:22.137172 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 20 08:48:22.137172 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 20 08:48:22.138181 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:22.138181 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:22.138181 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:22.138181 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:22.138181 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:22.138181 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:22.138181 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:22.138181 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:22.138181 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:22.138181 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:22.138181 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:22.138181 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:22.138181 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:22.138181 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:22.138181 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:22.138181 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:22.138181 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 20 08:48:22.139071 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 20 08:48:22.139071 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:22.139071 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:22.146728 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 173/348] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:48:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:22.146728 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93a3e291-836b-4137-aefc-9190cd73bd5f None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:22.210767 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-721579d5-f982-493f-a299-76abb984007c None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:22.245503 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-489179bb-658d-4861-9364-c1560f2a3886 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:22.246106 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-489179bb-658d-4861-9364-c1560f2a3886 None admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:22.246387 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-489179bb-658d-4861-9364-c1560f2a3886 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:22.246688 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-489179bb-658d-4861-9364-c1560f2a3886 None admin] PATH_INFO: `/v3/users` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:22.247326 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-489179bb-658d-4861-9364-c1560f2a3886 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:22.248773 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-489179bb-658d-4861-9364-c1560f2a3886 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:22.249652 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93a3e291-836b-4137-aefc-9190cd73bd5f None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:22.271482 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 176/349] 10.222.0.36 () {60 vars in 1132 bytes} [Thu Oct 20 08:48:22 2022] POST /identity/v3/users => generated 246 bytes in 185 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 08:48:22.275448 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-489179bb-658d-4861-9364-c1560f2a3886 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:22.288890 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d58ce564-bfe4-43de-b193-b57781dd6640 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:22.289683 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58ce564-bfe4-43de-b193-b57781dd6640 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:22.289962 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58ce564-bfe4-43de-b193-b57781dd6640 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:22.290228 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58ce564-bfe4-43de-b193-b57781dd6640 None admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:22.290981 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d58ce564-bfe4-43de-b193-b57781dd6640 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:22.291924 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d58ce564-bfe4-43de-b193-b57781dd6640 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:22.292767 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d58ce564-bfe4-43de-b193-b57781dd6640 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:22.312362 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 174/350] 10.222.0.36 () {56 vars in 1109 bytes} [Thu Oct 20 08:48:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:22.313357 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d58ce564-bfe4-43de-b193-b57781dd6640 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:22.381029 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:22.391517 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:22.435717 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-489179bb-658d-4861-9364-c1560f2a3886 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:22.443743 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d58ce564-bfe4-43de-b193-b57781dd6640 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:22.488566 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-114bd16e-7d0b-4449-bbff-5bbf9e87ab3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:22.489540 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-114bd16e-7d0b-4449-bbff-5bbf9e87ab3d None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:22.489910 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-114bd16e-7d0b-4449-bbff-5bbf9e87ab3d None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:22.490256 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-114bd16e-7d0b-4449-bbff-5bbf9e87ab3d None admin] PATH_INFO: `/v3/domains` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:22.491014 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-114bd16e-7d0b-4449-bbff-5bbf9e87ab3d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:22.491317 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-114bd16e-7d0b-4449-bbff-5bbf9e87ab3d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:22.493133 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-114bd16e-7d0b-4449-bbff-5bbf9e87ab3d None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:22.496050 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f9a4d5e-5782-4fb4-8fb4-76dea3b527c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:22.496996 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f9a4d5e-5782-4fb4-8fb4-76dea3b527c4 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:22.497420 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f9a4d5e-5782-4fb4-8fb4-76dea3b527c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:22.497834 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f9a4d5e-5782-4fb4-8fb4-76dea3b527c4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:22.509651 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f9a4d5e-5782-4fb4-8fb4-76dea3b527c4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:22.515532 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 177/351] 10.222.0.36 () {56 vars in 1109 bytes} [Thu Oct 20 08:48:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:22.515532 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-114bd16e-7d0b-4449-bbff-5bbf9e87ab3d None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:22.515775 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f9a4d5e-5782-4fb4-8fb4-76dea3b527c4 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:22.533138 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-1f9a4d5e-5782-4fb4-8fb4-76dea3b527c4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:22.533138 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:22.533138 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:22.533138 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:22.533138 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:22.533138 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:22.533138 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:22.533138 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:22.533138 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:22.533138 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:22.533138 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:22.533138 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:22.533138 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 20 08:48:22.533138 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 20 08:48:22.533138 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 20 08:48:22.533138 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 20 08:48:22.534328 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:22.534328 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:22.534328 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:22.534328 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:22.534328 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:22.534328 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:22.534328 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:22.534328 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:22.534328 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:22.534328 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:22.534328 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:22.534328 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:22.534328 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:22.534328 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:22.534328 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:22.534328 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:22.534328 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 20 08:48:22.535421 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 20 08:48:22.535421 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:22.535421 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:22.537316 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 175/352] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:48:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:22.538538 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f9a4d5e-5782-4fb4-8fb4-76dea3b527c4 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:22.656297 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-114bd16e-7d0b-4449-bbff-5bbf9e87ab3d None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:22.669731 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f9a4d5e-5782-4fb4-8fb4-76dea3b527c4 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:22.683250 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:22.685768 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:22.695582 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d0728e0-1971-4607-bc9e-5a53ae4dff4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:22.696185 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d0728e0-1971-4607-bc9e-5a53ae4dff4b None admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:22.696462 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d0728e0-1971-4607-bc9e-5a53ae4dff4b None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:22.696716 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d0728e0-1971-4607-bc9e-5a53ae4dff4b None admin] PATH_INFO: `/v3/users` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:22.697358 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d0728e0-1971-4607-bc9e-5a53ae4dff4b None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:22.698811 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d0728e0-1971-4607-bc9e-5a53ae4dff4b None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:22.705129 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb7132b9-175e-4527-b925-59fc6d4ab296 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:22.705919 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb7132b9-175e-4527-b925-59fc6d4ab296 None admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:22.706356 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb7132b9-175e-4527-b925-59fc6d4ab296 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:22.706816 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb7132b9-175e-4527-b925-59fc6d4ab296 None admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:22.707625 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb7132b9-175e-4527-b925-59fc6d4ab296 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:22.709887 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb7132b9-175e-4527-b925-59fc6d4ab296 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:22.730510 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 178/353] 10.222.0.36 () {60 vars in 1132 bytes} [Thu Oct 20 08:48:22 2022] POST /identity/v3/users => generated 248 bytes in 209 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 08:48:22.730510 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d0728e0-1971-4607-bc9e-5a53ae4dff4b None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:22.747299 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 176/354] 10.222.0.36 () {60 vars in 1132 bytes} [Thu Oct 20 08:48:22 2022] POST /identity/v3/users => generated 248 bytes in 209 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 08:48:22.749103 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b30ea8-981c-4576-8389-f3c56e042e67 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:22.749536 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b30ea8-981c-4576-8389-f3c56e042e67 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:22.750385 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b30ea8-981c-4576-8389-f3c56e042e67 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:22.752505 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 177/355] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:48:22 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:48:22.758316 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e826713-ab48-41f7-9114-52dec929eed1 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:22.758653 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e826713-ab48-41f7-9114-52dec929eed1 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:22.758920 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e826713-ab48-41f7-9114-52dec929eed1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:22.779608 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-4e826713-ab48-41f7-9114-52dec929eed1 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:48:22.780922 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-4e826713-ab48-41f7-9114-52dec929eed1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:48:22.854335 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d0728e0-1971-4607-bc9e-5a53ae4dff4b None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:22.859412 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 178/356] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:48:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 20 08:48:22.895331 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-604a2a56-a79d-45cc-bdeb-5c5f7b5c780e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:22.896142 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-604a2a56-a79d-45cc-bdeb-5c5f7b5c780e None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:22.896555 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-604a2a56-a79d-45cc-bdeb-5c5f7b5c780e None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:22.896834 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-604a2a56-a79d-45cc-bdeb-5c5f7b5c780e None admin] PATH_INFO: `/v3/domains` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:22.897598 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-604a2a56-a79d-45cc-bdeb-5c5f7b5c780e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:22.897907 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-604a2a56-a79d-45cc-bdeb-5c5f7b5c780e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:22.899409 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-604a2a56-a79d-45cc-bdeb-5c5f7b5c780e None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:22.915252 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 179/357] 10.222.0.36 () {56 vars in 1109 bytes} [Thu Oct 20 08:48:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:22.920493 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e826713-ab48-41f7-9114-52dec929eed1 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:23.046270 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:23.050780 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e826713-ab48-41f7-9114-52dec929eed1 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:23.089772 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3936f9d7-f1e0-45c7-91dd-509782794460 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:23.090415 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3936f9d7-f1e0-45c7-91dd-509782794460 None admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:23.090666 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3936f9d7-f1e0-45c7-91dd-509782794460 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:23.090940 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3936f9d7-f1e0-45c7-91dd-509782794460 None admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:23.091588 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3936f9d7-f1e0-45c7-91dd-509782794460 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:23.093015 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3936f9d7-f1e0-45c7-91dd-509782794460 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:23.118604 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 179/358] 10.222.0.36 () {60 vars in 1132 bytes} [Thu Oct 20 08:48:22 2022] POST /identity/v3/users => generated 249 bytes in 198 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 08:48:23.177248 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-604a2a56-a79d-45cc-bdeb-5c5f7b5c780e None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:23.248288 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:23.327812 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-604a2a56-a79d-45cc-bdeb-5c5f7b5c780e None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:23.372402 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6cfc869-8bd7-4e17-b2ed-0659626dc351 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:23.373087 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6cfc869-8bd7-4e17-b2ed-0659626dc351 None admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:23.373368 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6cfc869-8bd7-4e17-b2ed-0659626dc351 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:23.373636 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6cfc869-8bd7-4e17-b2ed-0659626dc351 None admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:23.374318 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6cfc869-8bd7-4e17-b2ed-0659626dc351 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:23.376385 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6cfc869-8bd7-4e17-b2ed-0659626dc351 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:23.387560 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.sql.core [None req-f6cfc869-8bd7-4e17-b2ed-0659626dc351 None admin] Conflict role: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Oct 20 08:48:23.387560 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Oct 20 08:48:23.387560 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [parameters: {'id': 'a9573b93debf4de98f6104039f7bb6e4', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Oct 20 08:48:23.387560 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81561) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 20 08:48:23.404821 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-f6cfc869-8bd7-4e17-b2ed-0659626dc351 None admin] Conflict occurred attempting to store role - Duplicate entry found with name anotherrole.: keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Oct 20 08:48:23.404821 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:23.404821 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 20 08:48:23.404821 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 20 08:48:23.404821 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 20 08:48:23.404821 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 20 08:48:23.404821 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 20 08:48:23.404821 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application result = self._query(query) Oct 20 08:48:23.404821 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 20 08:48:23.404821 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application conn.query(q) Oct 20 08:48:23.404821 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 20 08:48:23.404821 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 20 08:48:23.404821 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 20 08:48:23.404821 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application result.read() Oct 20 08:48:23.404821 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 20 08:48:23.405741 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 20 08:48:23.405741 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 20 08:48:23.405741 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 20 08:48:23.405741 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 20 08:48:23.405741 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 20 08:48:23.405741 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 20 08:48:23.405741 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 20 08:48:23.405741 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Oct 20 08:48:23.405741 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:48:23.405741 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 20 08:48:23.405741 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:48:23.405741 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:23.405741 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 20 08:48:23.405741 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 20 08:48:23.405741 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 33, in create_role Oct 20 08:48:23.405741 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return ref.to_dict() Oct 20 08:48:23.405741 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 20 08:48:23.405741 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application next(self.gen) Oct 20 08:48:23.406912 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Oct 20 08:48:23.406912 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application yield resource Oct 20 08:48:23.406912 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 20 08:48:23.406912 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application next(self.gen) Oct 20 08:48:23.406912 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Oct 20 08:48:23.406912 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application self.session.rollback() Oct 20 08:48:23.406912 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 20 08:48:23.406912 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application self.force_reraise() Oct 20 08:48:23.406912 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 20 08:48:23.406912 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise self.value Oct 20 08:48:23.406912 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Oct 20 08:48:23.406912 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 20 08:48:23.406912 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Oct 20 08:48:23.406912 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application session.commit() Oct 20 08:48:23.406912 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 20 08:48:23.406912 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 20 08:48:23.406912 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 20 08:48:23.407943 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application self._prepare_impl() Oct 20 08:48:23.407943 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 20 08:48:23.407943 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application self.session.flush() Oct 20 08:48:23.407943 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 20 08:48:23.407943 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application self._flush(objects) Oct 20 08:48:23.407943 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 20 08:48:23.407943 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 20 08:48:23.407943 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 20 08:48:23.407943 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application compat.raise_( Oct 20 08:48:23.407943 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 20 08:48:23.407943 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception Oct 20 08:48:23.407943 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 20 08:48:23.407943 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application flush_context.execute() Oct 20 08:48:23.407943 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 20 08:48:23.407943 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rec.execute(self) Oct 20 08:48:23.407943 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 20 08:48:23.407943 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 20 08:48:23.408843 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 20 08:48:23.408843 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 20 08:48:23.408843 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Oct 20 08:48:23.408843 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application c = connection._execute_20( Oct 20 08:48:23.408843 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 20 08:48:23.408843 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 20 08:48:23.408843 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 20 08:48:23.408843 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 20 08:48:23.408843 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 20 08:48:23.408843 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 20 08:48:23.408843 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 20 08:48:23.408843 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 20 08:48:23.408843 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 20 08:48:23.408843 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 20 08:48:23.408843 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 20 08:48:23.408843 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception Oct 20 08:48:23.409643 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 20 08:48:23.409643 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 20 08:48:23.409643 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 20 08:48:23.409643 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 20 08:48:23.409643 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 20 08:48:23.409643 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application result = self._query(query) Oct 20 08:48:23.409643 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 20 08:48:23.409643 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application conn.query(q) Oct 20 08:48:23.409643 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 20 08:48:23.409643 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 20 08:48:23.409643 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 20 08:48:23.409643 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application result.read() Oct 20 08:48:23.409643 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 20 08:48:23.409643 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 20 08:48:23.409643 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 20 08:48:23.409643 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 20 08:48:23.410430 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 20 08:48:23.410430 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 20 08:48:23.410430 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 20 08:48:23.410430 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 20 08:48:23.410430 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Oct 20 08:48:23.410430 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Oct 20 08:48:23.410430 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [parameters: {'id': 'a9573b93debf4de98f6104039f7bb6e4', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Oct 20 08:48:23.410430 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 20 08:48:23.410430 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:48:23.410430 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 20 08:48:23.410430 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:48:23.410430 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:23.410430 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:23.410430 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:23.410430 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:23.410430 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:23.412380 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:23.412380 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:23.412380 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:23.412380 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:23.412380 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:23.412380 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:23.412380 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 115, in post Oct 20 08:48:23.412380 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application ref = PROVIDERS.role_api.create_role( Oct 20 08:48:23.412380 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:23.412380 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:23.412380 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1298, in create_role Oct 20 08:48:23.412380 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application ret = self.driver.create_role(role_id, role) Oct 20 08:48:23.412380 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 20 08:48:23.412380 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 20 08:48:23.412380 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Oct 20 08:48:23.412380 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:48:23.413261 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 180/359] 10.222.0.36 () {60 vars in 1131 bytes} [Thu Oct 20 08:48:23 2022] POST /identity/v3/roles => generated 144 bytes in 230 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:23.414762 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3936f9d7-f1e0-45c7-91dd-509782794460 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:23.520113 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3936f9d7-f1e0-45c7-91dd-509782794460 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:23.555162 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7f2e8f3-bc84-4061-b251-7b99d16a7859 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:23.556150 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7f2e8f3-bc84-4061-b251-7b99d16a7859 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:23.556599 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7f2e8f3-bc84-4061-b251-7b99d16a7859 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:23.557046 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7f2e8f3-bc84-4061-b251-7b99d16a7859 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:23.567601 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7f2e8f3-bc84-4061-b251-7b99d16a7859 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 20 08:48:23.573171 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-c7f2e8f3-bc84-4061-b251-7b99d16a7859 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 20 08:48:23.573171 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:23.573171 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:23.573171 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:23.573171 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:23.573171 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:23.573171 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:23.573171 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:23.573171 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:23.573171 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:23.573171 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:23.573171 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:23.573171 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 20 08:48:23.573171 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 20 08:48:23.573171 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 20 08:48:23.573171 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 20 08:48:23.574063 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 20 08:48:23.574063 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 20 08:48:23.574063 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 20 08:48:23.574063 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 20 08:48:23.574063 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:23.574063 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:23.574063 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:23.574063 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:23.574063 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:23.574063 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:23.574063 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:23.574063 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:23.574063 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:23.574063 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:23.574063 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:23.574063 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:23.574947 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:23.574947 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:23.574947 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:23.574947 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:23.574947 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 20 08:48:23.574947 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 20 08:48:23.574947 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 20 08:48:23.574947 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 20 08:48:23.574947 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 20 08:48:23.574947 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 20 08:48:23.574947 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 20 08:48:23.574947 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:23.577198 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 180/360] 10.222.0.36 () {56 vars in 1114 bytes} [Thu Oct 20 08:48:23 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:23.582753 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6cfc869-8bd7-4e17-b2ed-0659626dc351 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:23.715000 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6cfc869-8bd7-4e17-b2ed-0659626dc351 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:23.722540 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:23.760119 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6e1d691-30f3-4ff2-8a5c-3aaad15149c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:23.760752 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e1d691-30f3-4ff2-8a5c-3aaad15149c7 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:23.761035 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e1d691-30f3-4ff2-8a5c-3aaad15149c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:23.761296 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e1d691-30f3-4ff2-8a5c-3aaad15149c7 None admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:23.761965 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6e1d691-30f3-4ff2-8a5c-3aaad15149c7 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:23.762262 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6e1d691-30f3-4ff2-8a5c-3aaad15149c7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:23.763668 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6e1d691-30f3-4ff2-8a5c-3aaad15149c7 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:23.772809 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 181/361] 10.222.0.36 () {56 vars in 1111 bytes} [Thu Oct 20 08:48:23 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:24.994993 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c6e813-eb3b-4e23-b27a-a1c3f21ac1e9 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:24.995544 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c6e813-eb3b-4e23-b27a-a1c3f21ac1e9 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:24.995544 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c6e813-eb3b-4e23-b27a-a1c3f21ac1e9 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:24.997130 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 181/362] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:48:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:48:25.002296 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-115b1824-8ecc-487c-ab80-9dba2adc0672 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:25.002825 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-115b1824-8ecc-487c-ab80-9dba2adc0672 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:25.003014 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-115b1824-8ecc-487c-ab80-9dba2adc0672 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:25.027807 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-115b1824-8ecc-487c-ab80-9dba2adc0672 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:48:25.029034 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-115b1824-8ecc-487c-ab80-9dba2adc0672 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:48:25.132448 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 182/363] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:48:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 20 08:48:25.485478 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c6e813-eb3b-4e23-b27a-a1c3f21ac1e9 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:25.497276 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74dc6180-75ef-48e5-a818-d220bddcb3a0 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:25.497576 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74dc6180-75ef-48e5-a818-d220bddcb3a0 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:25.497816 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74dc6180-75ef-48e5-a818-d220bddcb3a0 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:25.499859 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 183/364] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:48:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:48:25.505234 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:25.506635 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-091401ba-8b19-4416-8786-3cb2b6c0fb3f None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:25.506915 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-091401ba-8b19-4416-8786-3cb2b6c0fb3f None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:25.507182 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-091401ba-8b19-4416-8786-3cb2b6c0fb3f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:25.529469 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-091401ba-8b19-4416-8786-3cb2b6c0fb3f None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:48:25.530993 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-091401ba-8b19-4416-8786-3cb2b6c0fb3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:48:25.619214 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c6e813-eb3b-4e23-b27a-a1c3f21ac1e9 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:25.626574 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 184/365] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:48:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 20 08:48:25.627886 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6127f31-4cde-4c2e-8dcf-9e54a2a35beb None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:25.628172 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6127f31-4cde-4c2e-8dcf-9e54a2a35beb None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:25.628444 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6127f31-4cde-4c2e-8dcf-9e54a2a35beb None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:25.629911 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 185/366] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:48:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:48:25.636743 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d5f4d9e-c49f-4a38-aa0c-c22a95eea024 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:25.637011 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d5f4d9e-c49f-4a38-aa0c-c22a95eea024 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:25.637286 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d5f4d9e-c49f-4a38-aa0c-c22a95eea024 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:25.673083 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-2d5f4d9e-c49f-4a38-aa0c-c22a95eea024 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:48:25.674284 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-2d5f4d9e-c49f-4a38-aa0c-c22a95eea024 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:48:25.682003 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-978c3c8d-7180-42b6-8ca2-e53cfda7a5d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:25.682665 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-978c3c8d-7180-42b6-8ca2-e53cfda7a5d8 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:25.682946 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-978c3c8d-7180-42b6-8ca2-e53cfda7a5d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:25.683218 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-978c3c8d-7180-42b6-8ca2-e53cfda7a5d8 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:25.704224 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-978c3c8d-7180-42b6-8ca2-e53cfda7a5d8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 20 08:48:25.710324 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-978c3c8d-7180-42b6-8ca2-e53cfda7a5d8 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 20 08:48:25.710324 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:25.710324 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:25.710324 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:25.710324 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:25.710324 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:25.710324 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:25.710324 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:25.710324 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:25.710324 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:25.710324 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:25.710324 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:25.710324 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 20 08:48:25.710324 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 20 08:48:25.710324 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 20 08:48:25.710324 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 20 08:48:25.711255 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 20 08:48:25.711255 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 20 08:48:25.711255 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 20 08:48:25.711255 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 20 08:48:25.711255 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:25.711255 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:25.711255 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:25.711255 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:25.711255 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:25.711255 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:25.711255 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:25.711255 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:25.711255 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:25.711255 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:25.711255 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:25.711255 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:25.712072 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:25.712072 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:25.712072 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:25.712072 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:25.712072 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 20 08:48:25.712072 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 20 08:48:25.712072 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 20 08:48:25.712072 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 20 08:48:25.712072 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 20 08:48:25.712072 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 20 08:48:25.712072 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 20 08:48:25.712072 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:25.713369 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 182/367] 10.222.0.36 () {56 vars in 1102 bytes} [Thu Oct 20 08:48:25 2022] GET /identity/v3/roles/service => generated 85 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:25.719844 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-978c3c8d-7180-42b6-8ca2-e53cfda7a5d8 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:25.725959 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:25.815985 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 186/368] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:48:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 20 08:48:25.846859 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:25.873125 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-978c3c8d-7180-42b6-8ca2-e53cfda7a5d8 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:25.907102 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ab333f-a95b-4798-b699-e30f4d352736 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:25.907627 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ab333f-a95b-4798-b699-e30f4d352736 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:25.907900 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ab333f-a95b-4798-b699-e30f4d352736 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:25.908210 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ab333f-a95b-4798-b699-e30f4d352736 None admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:25.908867 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06ab333f-a95b-4798-b699-e30f4d352736 None admin] RBAC: Adding query filter params (name=service) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:25.909165 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06ab333f-a95b-4798-b699-e30f4d352736 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:25.910550 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06ab333f-a95b-4798-b699-e30f4d352736 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:25.912414 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d5f4d9e-c49f-4a38-aa0c-c22a95eea024 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:25.922762 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 183/369] 10.222.0.36 () {56 vars in 1103 bytes} [Thu Oct 20 08:48:25 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:25.928822 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ab333f-a95b-4798-b699-e30f4d352736 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:26.047181 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ab333f-a95b-4798-b699-e30f4d352736 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:26.059929 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d5f4d9e-c49f-4a38-aa0c-c22a95eea024 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:26.082828 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-480e0622-1e79-47d9-afb3-fdd794906daf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:26.083749 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480e0622-1e79-47d9-afb3-fdd794906daf None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:26.084026 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480e0622-1e79-47d9-afb3-fdd794906daf None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:26.084294 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480e0622-1e79-47d9-afb3-fdd794906daf None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:26.087370 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:26.096431 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-480e0622-1e79-47d9-afb3-fdd794906daf None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:26.097883 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-480e0622-1e79-47d9-afb3-fdd794906daf None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:26.117288 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57856638-f6ee-4cf1-8331-c04199be0c71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:26.117846 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-480e0622-1e79-47d9-afb3-fdd794906daf None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:26.117846 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:26.117846 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:26.117846 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:26.117846 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:26.117846 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:26.117846 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:26.117846 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:26.117846 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:26.117846 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:26.117846 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:26.117846 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:26.117846 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 20 08:48:26.117846 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 20 08:48:26.117846 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 20 08:48:26.117846 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 20 08:48:26.118849 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:26.118849 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:26.118849 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:26.118849 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:26.118849 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:26.118849 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:26.118849 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:26.118849 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:26.118849 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:26.118849 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:26.118849 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:26.118849 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:26.118849 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:26.118849 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:26.118849 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:26.118849 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:26.118849 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 20 08:48:26.119722 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 20 08:48:26.119722 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:26.119722 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:26.120718 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57856638-f6ee-4cf1-8331-c04199be0c71 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:26.121182 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57856638-f6ee-4cf1-8331-c04199be0c71 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:26.121628 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57856638-f6ee-4cf1-8331-c04199be0c71 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:26.123670 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 184/370] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:48:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:26.124908 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca2021a-7632-4242-868e-ee7a0c4e6094 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:26.125188 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca2021a-7632-4242-868e-ee7a0c4e6094 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:26.125460 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca2021a-7632-4242-868e-ee7a0c4e6094 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:26.126989 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 185/371] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:48:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:48:26.127691 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ca2021a-7632-4242-868e-ee7a0c4e6094 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:26.136293 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57856638-f6ee-4cf1-8331-c04199be0c71 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 20 08:48:26.143058 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-57856638-f6ee-4cf1-8331-c04199be0c71 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 20 08:48:26.143058 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:26.143058 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:26.143058 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:26.143058 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:26.143058 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:26.143058 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:26.143058 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:26.143058 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:26.143058 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:26.143058 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:26.143058 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:26.143058 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 20 08:48:26.143058 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 20 08:48:26.143058 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 20 08:48:26.143058 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 20 08:48:26.144158 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 20 08:48:26.144158 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 20 08:48:26.144158 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 20 08:48:26.144158 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 20 08:48:26.144158 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:26.144158 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:26.144158 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:26.144158 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:26.144158 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:26.144158 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:26.144158 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:26.144158 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:26.144158 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:26.144158 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:26.144158 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:26.144158 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:26.145012 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:26.145012 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:26.145012 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:26.145012 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:26.145012 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 20 08:48:26.145012 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 20 08:48:26.145012 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 20 08:48:26.145012 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 20 08:48:26.145012 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 20 08:48:26.145012 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 20 08:48:26.145012 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 20 08:48:26.145012 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:48:26.147406 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 187/372] 10.222.0.36 () {56 vars in 1102 bytes} [Thu Oct 20 08:48:25 2022] GET /identity/v3/roles/service => generated 85 bytes in 236 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 20 08:48:26.148709 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a8c8c3-633d-4ced-9d8f-6ed9ee36b7b7 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:26.148994 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a8c8c3-633d-4ced-9d8f-6ed9ee36b7b7 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:26.149265 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a8c8c3-633d-4ced-9d8f-6ed9ee36b7b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:26.167717 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-87a8c8c3-633d-4ced-9d8f-6ed9ee36b7b7 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:48:26.168901 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-87a8c8c3-633d-4ced-9d8f-6ed9ee36b7b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:48:26.183508 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:26.238586 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 188/373] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:48:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 20 08:48:26.239432 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87a8c8c3-633d-4ced-9d8f-6ed9ee36b7b7 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:26.270348 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ca2021a-7632-4242-868e-ee7a0c4e6094 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:26.318808 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7dc1916-7203-47dc-9456-b3980c3cfdc0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:26.319679 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7dc1916-7203-47dc-9456-b3980c3cfdc0 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:26.320105 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7dc1916-7203-47dc-9456-b3980c3cfdc0 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:26.320521 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7dc1916-7203-47dc-9456-b3980c3cfdc0 None admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:26.321303 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7dc1916-7203-47dc-9456-b3980c3cfdc0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:26.321764 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7dc1916-7203-47dc-9456-b3980c3cfdc0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:26.323836 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7dc1916-7203-47dc-9456-b3980c3cfdc0 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:26.344816 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 186/374] 10.222.0.36 () {56 vars in 1109 bytes} [Thu Oct 20 08:48:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:26.345696 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7dc1916-7203-47dc-9456-b3980c3cfdc0 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:26.358139 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87a8c8c3-633d-4ced-9d8f-6ed9ee36b7b7 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:26.398659 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:26.406664 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f417147-1c36-4a9e-b03a-a48e3c7fcf8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:26.407543 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f417147-1c36-4a9e-b03a-a48e3c7fcf8e None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:26.408001 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f417147-1c36-4a9e-b03a-a48e3c7fcf8e None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:26.408470 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f417147-1c36-4a9e-b03a-a48e3c7fcf8e None admin] PATH_INFO: `/v3/roles/service` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:26.419913 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f417147-1c36-4a9e-b03a-a48e3c7fcf8e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 20 08:48:26.424968 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-9f417147-1c36-4a9e-b03a-a48e3c7fcf8e None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 20 08:48:26.424968 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:26.424968 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:26.424968 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:26.424968 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:26.424968 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:26.424968 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:26.424968 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:26.424968 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:26.424968 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:26.424968 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:26.424968 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:26.424968 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 20 08:48:26.424968 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 20 08:48:26.424968 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 20 08:48:26.424968 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 20 08:48:26.425933 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 20 08:48:26.425933 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 20 08:48:26.425933 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 20 08:48:26.425933 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 20 08:48:26.425933 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:26.425933 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:26.425933 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:26.425933 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:26.425933 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:26.425933 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:26.425933 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:26.425933 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:26.425933 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:26.425933 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:26.425933 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:26.425933 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:26.426954 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:26.426954 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:26.426954 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:26.426954 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:26.426954 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 20 08:48:26.426954 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 20 08:48:26.426954 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 20 08:48:26.426954 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 20 08:48:26.426954 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 20 08:48:26.426954 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 20 08:48:26.426954 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 20 08:48:26.426954 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:48:26.429356 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 189/375] 10.222.0.36 () {56 vars in 1102 bytes} [Thu Oct 20 08:48:26 2022] GET /identity/v3/roles/service => generated 85 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:26.430406 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f417147-1c36-4a9e-b03a-a48e3c7fcf8e None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:26.445114 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:26.466729 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7dc1916-7203-47dc-9456-b3980c3cfdc0 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:26.504005 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f54a661-4cbc-447f-b1d7-e9f889e351c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:26.505040 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f54a661-4cbc-447f-b1d7-e9f889e351c6 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:26.506259 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f54a661-4cbc-447f-b1d7-e9f889e351c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:26.506772 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f54a661-4cbc-447f-b1d7-e9f889e351c6 None admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:26.507692 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f54a661-4cbc-447f-b1d7-e9f889e351c6 None admin] RBAC: Adding query filter params (name=service) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:26.508225 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f54a661-4cbc-447f-b1d7-e9f889e351c6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:26.510005 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f54a661-4cbc-447f-b1d7-e9f889e351c6 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:26.519320 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 187/376] 10.222.0.36 () {56 vars in 1103 bytes} [Thu Oct 20 08:48:26 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:26.526095 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f54a661-4cbc-447f-b1d7-e9f889e351c6 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:26.557960 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f417147-1c36-4a9e-b03a-a48e3c7fcf8e None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:26.593674 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc3e8ad-ea76-4907-9a03-fbea37809d5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:26.594298 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc3e8ad-ea76-4907-9a03-fbea37809d5d None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:26.594604 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc3e8ad-ea76-4907-9a03-fbea37809d5d None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:26.594875 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc3e8ad-ea76-4907-9a03-fbea37809d5d None admin] PATH_INFO: `/v3/users/nova` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:26.602511 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cc3e8ad-ea76-4907-9a03-fbea37809d5d None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:26.603962 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cc3e8ad-ea76-4907-9a03-fbea37809d5d None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:26.616776 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-0cc3e8ad-ea76-4907-9a03-fbea37809d5d None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 20 08:48:26.616776 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:26.616776 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:26.616776 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:26.616776 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:26.616776 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:26.616776 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:26.616776 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:26.616776 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:26.616776 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:26.616776 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:26.616776 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:26.616776 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 20 08:48:26.616776 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 20 08:48:26.616776 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 20 08:48:26.616776 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 20 08:48:26.617751 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:26.617751 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:26.617751 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 20 08:48:26.617751 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 20 08:48:26.617751 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 20 08:48:26.617751 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 20 08:48:26.617751 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:26.617751 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:26.617751 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:26.617751 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:26.617751 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:26.617751 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:26.617751 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:26.617751 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:26.617751 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:26.617751 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:26.617751 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:26.617751 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:26.619204 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:26.619204 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:26.619204 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 20 08:48:26.619204 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 20 08:48:26.619204 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 20 08:48:26.619204 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 20 08:48:26.619204 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 20 08:48:26.619204 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 20 08:48:26.619204 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 20 08:48:26.619204 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:48:26.619204 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 190/377] 10.222.0.36 () {56 vars in 1093 bytes} [Thu Oct 20 08:48:26 2022] GET /identity/v3/users/nova => generated 82 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:26.619204 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc3e8ad-ea76-4907-9a03-fbea37809d5d None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:26.646613 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f54a661-4cbc-447f-b1d7-e9f889e351c6 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:26.656988 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:26.680074 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef10eea-158c-4dea-8403-f75ab18b8ce6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:26.680596 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef10eea-158c-4dea-8403-f75ab18b8ce6 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:26.680867 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef10eea-158c-4dea-8403-f75ab18b8ce6 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:26.681127 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef10eea-158c-4dea-8403-f75ab18b8ce6 None admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:26.681768 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eef10eea-158c-4dea-8403-f75ab18b8ce6 None admin] RBAC: Adding query filter params (name=service) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:26.682061 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eef10eea-158c-4dea-8403-f75ab18b8ce6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:26.683444 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eef10eea-158c-4dea-8403-f75ab18b8ce6 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:26.688079 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:26.691076 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 188/378] 10.222.0.36 () {56 vars in 1103 bytes} [Thu Oct 20 08:48:26 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:26.691842 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef10eea-158c-4dea-8403-f75ab18b8ce6 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:26.722763 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc3e8ad-ea76-4907-9a03-fbea37809d5d None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:26.756725 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33cdb98f-bb7e-4d6d-bb45-ac7196f65963 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:26.757354 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33cdb98f-bb7e-4d6d-bb45-ac7196f65963 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:26.757632 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33cdb98f-bb7e-4d6d-bb45-ac7196f65963 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:26.757906 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33cdb98f-bb7e-4d6d-bb45-ac7196f65963 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:26.768428 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33cdb98f-bb7e-4d6d-bb45-ac7196f65963 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:26.769902 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33cdb98f-bb7e-4d6d-bb45-ac7196f65963 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:26.784400 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-33cdb98f-bb7e-4d6d-bb45-ac7196f65963 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:26.784400 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:26.784400 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:26.784400 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:26.784400 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:26.784400 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:26.784400 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:26.784400 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:26.784400 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:26.784400 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:26.784400 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:26.784400 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:26.784400 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 20 08:48:26.784400 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 20 08:48:26.784400 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 20 08:48:26.784400 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 20 08:48:26.785560 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:26.785560 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:26.785560 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:26.785560 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:26.785560 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:26.785560 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:26.785560 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:26.785560 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:26.785560 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:26.785560 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:26.785560 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:26.785560 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:26.785560 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:26.785560 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:26.785560 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:26.785560 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:26.785560 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 20 08:48:26.786747 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 20 08:48:26.786747 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:26.786747 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:48:26.786747 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 191/379] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:48:26 2022] GET /identity/v3/domains/Default => generated 87 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:26.786747 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33cdb98f-bb7e-4d6d-bb45-ac7196f65963 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:26.798963 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef10eea-158c-4dea-8403-f75ab18b8ce6 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:26.831828 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b1fd9d-dec8-469f-a22f-66201693d780 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:26.832531 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b1fd9d-dec8-469f-a22f-66201693d780 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:26.832827 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b1fd9d-dec8-469f-a22f-66201693d780 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:26.833347 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b1fd9d-dec8-469f-a22f-66201693d780 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:26.843833 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36b1fd9d-dec8-469f-a22f-66201693d780 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 20 08:48:26.849164 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-36b1fd9d-dec8-469f-a22f-66201693d780 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 20 08:48:26.849164 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:26.849164 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:26.849164 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:26.849164 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:26.849164 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:26.849164 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:26.849164 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:26.849164 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:26.849164 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:26.849164 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:26.849164 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:26.849164 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 20 08:48:26.849164 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 20 08:48:26.849164 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 20 08:48:26.849164 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 20 08:48:26.850262 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 20 08:48:26.850262 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 20 08:48:26.850262 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 20 08:48:26.850262 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 20 08:48:26.850262 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:26.850262 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:26.850262 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:26.850262 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:26.850262 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:26.850262 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:26.850262 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:26.850262 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:26.850262 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:26.850262 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:26.850262 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:26.850262 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:26.851259 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:26.851259 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:26.851259 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:26.851259 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:26.851259 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 20 08:48:26.851259 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 20 08:48:26.851259 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 20 08:48:26.851259 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 20 08:48:26.851259 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 20 08:48:26.851259 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 20 08:48:26.851259 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 20 08:48:26.851259 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:26.851259 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 189/380] 10.222.0.36 () {56 vars in 1102 bytes} [Thu Oct 20 08:48:26 2022] GET /identity/v3/roles/service => generated 85 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:26.851919 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8701d01-9fd2-4b08-bf91-1f8f37a2127b None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:26.851919 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8701d01-9fd2-4b08-bf91-1f8f37a2127b None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:26.851919 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8701d01-9fd2-4b08-bf91-1f8f37a2127b None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:26.853134 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 190/381] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:48:26 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:48:26.853827 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8701d01-9fd2-4b08-bf91-1f8f37a2127b None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:26.885385 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:26.888816 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33cdb98f-bb7e-4d6d-bb45-ac7196f65963 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:26.908659 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:26.922670 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cf989cb-3291-4564-bf61-fee3ba9a0083 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:26.923271 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf989cb-3291-4564-bf61-fee3ba9a0083 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:26.923548 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf989cb-3291-4564-bf61-fee3ba9a0083 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:26.923815 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf989cb-3291-4564-bf61-fee3ba9a0083 None admin] PATH_INFO: `/v3/users` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:26.924530 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cf989cb-3291-4564-bf61-fee3ba9a0083 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:26.924828 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cf989cb-3291-4564-bf61-fee3ba9a0083 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:26.926243 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cf989cb-3291-4564-bf61-fee3ba9a0083 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:26.953415 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8701d01-9fd2-4b08-bf91-1f8f37a2127b None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:26.962599 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 192/382] 10.222.0.36 () {56 vars in 1133 bytes} [Thu Oct 20 08:48:26 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:26.963320 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cf989cb-3291-4564-bf61-fee3ba9a0083 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:26.990846 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42d56939-d624-4a4a-a892-abc32ab67eb1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:26.991466 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d56939-d624-4a4a-a892-abc32ab67eb1 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:26.991731 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d56939-d624-4a4a-a892-abc32ab67eb1 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:26.991995 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d56939-d624-4a4a-a892-abc32ab67eb1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:27.002438 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42d56939-d624-4a4a-a892-abc32ab67eb1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:27.003889 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42d56939-d624-4a4a-a892-abc32ab67eb1 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:27.018337 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-42d56939-d624-4a4a-a892-abc32ab67eb1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:27.018337 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:27.018337 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:27.018337 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:27.018337 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:27.018337 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:27.018337 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:27.018337 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:27.018337 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:27.018337 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:27.018337 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:27.018337 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:27.018337 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 20 08:48:27.018337 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 20 08:48:27.018337 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 20 08:48:27.018337 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 20 08:48:27.019966 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:27.019966 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:27.019966 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:27.019966 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:27.019966 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:27.019966 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:27.019966 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:27.019966 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:27.019966 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:27.019966 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:27.019966 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:27.019966 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:27.019966 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:27.019966 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:27.019966 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:27.019966 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:27.019966 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 20 08:48:27.021959 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 20 08:48:27.021959 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:27.021959 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:27.021959 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 191/383] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:48:26 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:27.023255 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42d56939-d624-4a4a-a892-abc32ab67eb1 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:27.061600 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cf989cb-3291-4564-bf61-fee3ba9a0083 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:27.095790 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cfec181-65b2-4a1c-b8c3-b38aecfedc82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:27.096393 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cfec181-65b2-4a1c-b8c3-b38aecfedc82 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:27.096668 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cfec181-65b2-4a1c-b8c3-b38aecfedc82 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:27.096937 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cfec181-65b2-4a1c-b8c3-b38aecfedc82 None admin] PATH_INFO: `/v3/domains` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:27.097568 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cfec181-65b2-4a1c-b8c3-b38aecfedc82 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:27.097863 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cfec181-65b2-4a1c-b8c3-b38aecfedc82 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:27.099434 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cfec181-65b2-4a1c-b8c3-b38aecfedc82 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:27.107074 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:27.110936 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 193/384] 10.222.0.36 () {56 vars in 1109 bytes} [Thu Oct 20 08:48:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:27.112090 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cada292e-f8bf-4c66-b3d3-2a453ec8c15f None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:27.112370 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cada292e-f8bf-4c66-b3d3-2a453ec8c15f None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:27.112638 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cada292e-f8bf-4c66-b3d3-2a453ec8c15f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:27.124460 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42d56939-d624-4a4a-a892-abc32ab67eb1 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:27.131310 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-cada292e-f8bf-4c66-b3d3-2a453ec8c15f None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:48:27.132519 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-cada292e-f8bf-4c66-b3d3-2a453ec8c15f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:48:27.157532 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5184bbd-0e1d-48be-9f3a-4afa96772e2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:27.158119 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5184bbd-0e1d-48be-9f3a-4afa96772e2e None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:27.158393 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5184bbd-0e1d-48be-9f3a-4afa96772e2e None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:27.158699 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5184bbd-0e1d-48be-9f3a-4afa96772e2e None admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:27.159341 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5184bbd-0e1d-48be-9f3a-4afa96772e2e None admin] RBAC: Adding query filter params (name=service) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:27.159633 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5184bbd-0e1d-48be-9f3a-4afa96772e2e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:27.160734 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:27.161095 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5184bbd-0e1d-48be-9f3a-4afa96772e2e None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:27.168993 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 192/385] 10.222.0.36 () {56 vars in 1103 bytes} [Thu Oct 20 08:48:27 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:27.169769 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5184bbd-0e1d-48be-9f3a-4afa96772e2e None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:27.199007 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 194/386] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:48:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 20 08:48:27.200110 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cada292e-f8bf-4c66-b3d3-2a453ec8c15f None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:27.272452 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5184bbd-0e1d-48be-9f3a-4afa96772e2e None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:27.307308 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cada292e-f8bf-4c66-b3d3-2a453ec8c15f None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:27.309080 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-601f49ad-433c-4cd5-9343-417d9f20ee30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:27.310125 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-601f49ad-433c-4cd5-9343-417d9f20ee30 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:27.310125 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-601f49ad-433c-4cd5-9343-417d9f20ee30 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:27.310305 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-601f49ad-433c-4cd5-9343-417d9f20ee30 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:27.321440 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-601f49ad-433c-4cd5-9343-417d9f20ee30 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:27.322915 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-601f49ad-433c-4cd5-9343-417d9f20ee30 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:27.330394 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:27.339111 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-601f49ad-433c-4cd5-9343-417d9f20ee30 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:27.339111 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:27.339111 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:27.339111 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:27.339111 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:27.339111 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:27.339111 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:27.339111 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:27.339111 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:27.339111 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:27.339111 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:27.339111 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:27.339111 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 20 08:48:27.339111 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 20 08:48:27.339111 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 20 08:48:27.339111 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 20 08:48:27.340225 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:27.340225 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:27.340225 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:27.340225 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:27.340225 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:27.340225 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:27.340225 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:27.340225 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:27.340225 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:27.340225 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:27.340225 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:27.340225 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:27.340225 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:27.340225 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:27.340225 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:27.340225 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:27.340225 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 20 08:48:27.341106 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 20 08:48:27.341106 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:27.341106 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:27.346215 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-310083dc-feec-445c-8b67-95a2b297260e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:27.346852 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310083dc-feec-445c-8b67-95a2b297260e None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:27.347130 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310083dc-feec-445c-8b67-95a2b297260e None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:27.347402 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310083dc-feec-445c-8b67-95a2b297260e None admin] PATH_INFO: `/v3/domains` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:27.348104 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 193/387] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:48:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:27.348208 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-310083dc-feec-445c-8b67-95a2b297260e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:27.348405 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-310083dc-feec-445c-8b67-95a2b297260e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:27.348704 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-601f49ad-433c-4cd5-9343-417d9f20ee30 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:27.349805 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-310083dc-feec-445c-8b67-95a2b297260e None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:27.361539 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 195/388] 10.222.0.36 () {56 vars in 1109 bytes} [Thu Oct 20 08:48:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:27.362330 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-310083dc-feec-445c-8b67-95a2b297260e None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:27.391942 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:27.468078 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-601f49ad-433c-4cd5-9343-417d9f20ee30 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:27.468550 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-310083dc-feec-445c-8b67-95a2b297260e None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:27.500960 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bbeca25-7ec3-45f7-b359-cbfb7909166f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:27.501567 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bbeca25-7ec3-45f7-b359-cbfb7909166f None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:27.501683 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c9f3fea-5404-4b3d-8ded-421e14289281 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:27.501841 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bbeca25-7ec3-45f7-b359-cbfb7909166f None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:27.502105 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bbeca25-7ec3-45f7-b359-cbfb7909166f None admin] PATH_INFO: `/v3/users/glance` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:27.502277 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c9f3fea-5404-4b3d-8ded-421e14289281 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:27.502641 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c9f3fea-5404-4b3d-8ded-421e14289281 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:27.502853 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c9f3fea-5404-4b3d-8ded-421e14289281 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:27.509300 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bbeca25-7ec3-45f7-b359-cbfb7909166f None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:27.510757 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bbeca25-7ec3-45f7-b359-cbfb7909166f None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:27.513148 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c9f3fea-5404-4b3d-8ded-421e14289281 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:27.514593 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c9f3fea-5404-4b3d-8ded-421e14289281 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:27.524099 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-1bbeca25-7ec3-45f7-b359-cbfb7909166f None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Oct 20 08:48:27.524099 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:27.524099 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:27.524099 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:27.524099 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:27.524099 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:27.524099 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:27.524099 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:27.524099 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:27.524099 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:27.524099 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:27.524099 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:27.524099 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 20 08:48:27.524099 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 20 08:48:27.524099 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 20 08:48:27.524099 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 20 08:48:27.524983 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:27.524983 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:27.524983 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 20 08:48:27.524983 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 20 08:48:27.524983 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 20 08:48:27.524983 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 20 08:48:27.524983 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:27.524983 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:27.524983 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:27.524983 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:27.524983 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:27.524983 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:27.524983 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:27.524983 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:27.524983 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:27.524983 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:27.524983 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:27.524983 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:27.525915 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:27.525915 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:27.525915 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 20 08:48:27.525915 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 20 08:48:27.525915 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 20 08:48:27.525915 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 20 08:48:27.525915 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 20 08:48:27.525915 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 20 08:48:27.525915 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Oct 20 08:48:27.525915 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:27.525915 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 194/389] 10.222.0.36 () {56 vars in 1099 bytes} [Thu Oct 20 08:48:27 2022] GET /identity/v3/users/glance => generated 84 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:27.525915 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bbeca25-7ec3-45f7-b359-cbfb7909166f None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:27.528250 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-7c9f3fea-5404-4b3d-8ded-421e14289281 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:27.528250 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:27.528250 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:27.528250 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:27.528250 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:27.528250 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:27.528250 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:27.528250 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:27.528250 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:27.528250 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:27.528250 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:27.528250 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:27.528250 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 20 08:48:27.528250 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 20 08:48:27.528250 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 20 08:48:27.528250 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 20 08:48:27.529269 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:27.529269 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:27.529269 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:27.529269 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:27.529269 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:27.529269 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:27.529269 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:27.529269 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:27.529269 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:27.529269 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:27.529269 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:27.529269 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:27.529269 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:27.529269 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:27.529269 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:27.529269 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:27.529269 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 20 08:48:27.530898 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 20 08:48:27.530898 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:27.530898 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:48:27.530898 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 196/390] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:48:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:27.530898 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c9f3fea-5404-4b3d-8ded-421e14289281 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:27.590343 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:27.609861 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:27.628997 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c9f3fea-5404-4b3d-8ded-421e14289281 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:27.631950 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bbeca25-7ec3-45f7-b359-cbfb7909166f None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:27.661002 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22fc7789-8403-402b-8697-f363253e884a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:27.662131 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22fc7789-8403-402b-8697-f363253e884a None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:27.662409 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22fc7789-8403-402b-8697-f363253e884a None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:27.662707 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22fc7789-8403-402b-8697-f363253e884a None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:27.664598 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e52174ea-dbc3-4b21-b4c6-070fc30742d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:27.665178 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52174ea-dbc3-4b21-b4c6-070fc30742d6 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:27.665454 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52174ea-dbc3-4b21-b4c6-070fc30742d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:27.665716 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52174ea-dbc3-4b21-b4c6-070fc30742d6 None admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:27.666344 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e52174ea-dbc3-4b21-b4c6-070fc30742d6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:27.666669 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e52174ea-dbc3-4b21-b4c6-070fc30742d6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:27.668031 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e52174ea-dbc3-4b21-b4c6-070fc30742d6 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:27.670255 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22fc7789-8403-402b-8697-f363253e884a None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:27.671736 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22fc7789-8403-402b-8697-f363253e884a None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:27.679916 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 195/391] 10.222.0.36 () {56 vars in 1109 bytes} [Thu Oct 20 08:48:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:27.680695 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e52174ea-dbc3-4b21-b4c6-070fc30742d6 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:27.683257 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-22fc7789-8403-402b-8697-f363253e884a None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 20 08:48:27.683257 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:27.683257 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:27.683257 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:27.683257 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:27.683257 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:27.683257 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:27.683257 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:27.683257 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:27.683257 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:27.683257 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:27.683257 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:27.683257 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 20 08:48:27.683257 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 20 08:48:27.683257 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 20 08:48:27.683257 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 20 08:48:27.684277 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:27.684277 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:27.684277 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 20 08:48:27.684277 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 20 08:48:27.684277 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 20 08:48:27.684277 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 20 08:48:27.684277 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:27.684277 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:27.684277 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:27.684277 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:27.684277 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:27.684277 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:27.684277 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:27.684277 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:27.684277 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:27.684277 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:27.684277 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:27.684277 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:27.685183 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:27.685183 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:27.685183 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 20 08:48:27.685183 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 20 08:48:27.685183 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 20 08:48:27.685183 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 20 08:48:27.685183 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 20 08:48:27.685183 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 20 08:48:27.685183 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 20 08:48:27.685183 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:48:27.685183 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 197/392] 10.222.0.36 () {56 vars in 1099 bytes} [Thu Oct 20 08:48:27 2022] GET /identity/v3/users/cinder => generated 84 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:27.685183 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22fc7789-8403-402b-8697-f363253e884a None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:27.780052 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e52174ea-dbc3-4b21-b4c6-070fc30742d6 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:27.781253 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22fc7789-8403-402b-8697-f363253e884a None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:27.798872 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:27.815463 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b9d386b-38ba-41fb-90f7-e275e06bc4d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:27.816052 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9d386b-38ba-41fb-90f7-e275e06bc4d5 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:27.816168 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-806a0cd9-987d-442d-9957-f3c985d00384 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:27.816326 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9d386b-38ba-41fb-90f7-e275e06bc4d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:27.816593 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9d386b-38ba-41fb-90f7-e275e06bc4d5 None admin] PATH_INFO: `/v3/users` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:27.816723 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-806a0cd9-987d-442d-9957-f3c985d00384 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:27.817001 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-806a0cd9-987d-442d-9957-f3c985d00384 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:27.817268 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-806a0cd9-987d-442d-9957-f3c985d00384 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:27.817354 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b9d386b-38ba-41fb-90f7-e275e06bc4d5 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:27.817596 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b9d386b-38ba-41fb-90f7-e275e06bc4d5 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:27.819134 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b9d386b-38ba-41fb-90f7-e275e06bc4d5 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:27.827068 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-806a0cd9-987d-442d-9957-f3c985d00384 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:27.827843 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:27.828512 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-806a0cd9-987d-442d-9957-f3c985d00384 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:27.836893 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 198/393] 10.222.0.36 () {56 vars in 1137 bytes} [Thu Oct 20 08:48:27 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:27.837868 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b9d386b-38ba-41fb-90f7-e275e06bc4d5 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:27.844965 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-806a0cd9-987d-442d-9957-f3c985d00384 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:27.844965 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:27.844965 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:27.844965 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:27.844965 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:27.844965 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:27.844965 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:27.844965 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:27.844965 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:27.844965 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:27.844965 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:27.844965 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:27.844965 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 20 08:48:27.844965 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 20 08:48:27.844965 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 20 08:48:27.844965 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 20 08:48:27.846062 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:27.846062 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:27.846062 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:27.846062 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:27.846062 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:27.846062 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:27.846062 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:27.846062 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:27.846062 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:27.846062 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:27.846062 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:27.846062 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:27.846062 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:27.846062 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:27.846062 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:27.846062 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:27.846062 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 20 08:48:27.847198 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 20 08:48:27.847198 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:27.847198 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:27.847198 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 196/394] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:48:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:27.847198 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-806a0cd9-987d-442d-9957-f3c985d00384 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:27.937762 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b9d386b-38ba-41fb-90f7-e275e06bc4d5 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:27.944651 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-806a0cd9-987d-442d-9957-f3c985d00384 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:27.971359 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b785d774-ac0c-4ced-a23d-db5a39f74987 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:27.971959 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b785d774-ac0c-4ced-a23d-db5a39f74987 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:27.972235 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b785d774-ac0c-4ced-a23d-db5a39f74987 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:27.972501 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b785d774-ac0c-4ced-a23d-db5a39f74987 None admin] PATH_INFO: `/v3/domains` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:27.973131 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b785d774-ac0c-4ced-a23d-db5a39f74987 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:27.973425 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b785d774-ac0c-4ced-a23d-db5a39f74987 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:27.974876 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b785d774-ac0c-4ced-a23d-db5a39f74987 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:27.977689 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-619c7867-1dd3-4b44-842c-6e06184af0e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:27.978303 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-619c7867-1dd3-4b44-842c-6e06184af0e2 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:27.978616 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-619c7867-1dd3-4b44-842c-6e06184af0e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:27.978888 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-619c7867-1dd3-4b44-842c-6e06184af0e2 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:27.985454 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-619c7867-1dd3-4b44-842c-6e06184af0e2 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:27.985711 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 199/395] 10.222.0.36 () {56 vars in 1109 bytes} [Thu Oct 20 08:48:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:27.986513 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b785d774-ac0c-4ced-a23d-db5a39f74987 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:27.987747 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-619c7867-1dd3-4b44-842c-6e06184af0e2 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:28.000254 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-619c7867-1dd3-4b44-842c-6e06184af0e2 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 20 08:48:28.000254 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:28.000254 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:28.000254 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:28.000254 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:28.000254 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:28.000254 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:28.000254 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:28.000254 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:28.000254 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:28.000254 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:28.000254 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:28.000254 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 20 08:48:28.000254 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 20 08:48:28.000254 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 20 08:48:28.001286 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 20 08:48:28.001286 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:28.001286 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:28.001286 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:28.001286 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:28.001286 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:28.001286 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:28.001286 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:28.001286 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:28.001286 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:28.001286 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:28.001286 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:28.001286 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:28.001286 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:28.001286 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:28.001286 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:28.001286 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:28.002218 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 20 08:48:28.002218 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 20 08:48:28.002218 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 20 08:48:28.002218 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 20 08:48:28.002218 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 20 08:48:28.002218 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 20 08:48:28.002218 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 20 08:48:28.002218 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:28.002218 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 197/396] 10.222.0.36 () {56 vars in 1111 bytes} [Thu Oct 20 08:48:27 2022] GET /identity/v3/projects/service => generated 88 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:28.002908 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-619c7867-1dd3-4b44-842c-6e06184af0e2 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:28.032166 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:28.046853 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:28.089955 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b785d774-ac0c-4ced-a23d-db5a39f74987 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:28.100489 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-619c7867-1dd3-4b44-842c-6e06184af0e2 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:28.123016 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6abd0696-5455-4141-b580-e77767c3f79e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:28.123613 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6abd0696-5455-4141-b580-e77767c3f79e None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:28.123887 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6abd0696-5455-4141-b580-e77767c3f79e None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:28.124152 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6abd0696-5455-4141-b580-e77767c3f79e None admin] PATH_INFO: `/v3/users` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:28.124867 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6abd0696-5455-4141-b580-e77767c3f79e None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:28.125167 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6abd0696-5455-4141-b580-e77767c3f79e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:28.126634 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6abd0696-5455-4141-b580-e77767c3f79e None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:28.135691 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e64c2c77-99b4-462e-9e17-08101f725bea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:28.136309 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64c2c77-99b4-462e-9e17-08101f725bea None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:28.136586 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64c2c77-99b4-462e-9e17-08101f725bea None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:28.136855 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64c2c77-99b4-462e-9e17-08101f725bea None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:28.144104 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 200/397] 10.222.0.36 () {56 vars in 1137 bytes} [Thu Oct 20 08:48:27 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:28.144952 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6abd0696-5455-4141-b580-e77767c3f79e None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:28.147848 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e64c2c77-99b4-462e-9e17-08101f725bea None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:28.149328 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e64c2c77-99b4-462e-9e17-08101f725bea None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:28.163996 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-e64c2c77-99b4-462e-9e17-08101f725bea None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:28.163996 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:28.163996 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:28.163996 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:28.163996 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:28.163996 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:28.163996 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:28.163996 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:28.163996 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:28.163996 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:28.163996 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:28.163996 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:28.163996 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 20 08:48:28.163996 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 20 08:48:28.163996 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 20 08:48:28.163996 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 20 08:48:28.165088 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:28.165088 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:28.165088 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:28.165088 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:28.165088 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:28.165088 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:28.165088 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:28.165088 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:28.165088 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:28.165088 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:28.165088 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:28.165088 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:28.165088 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:28.165088 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:28.165088 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:28.165088 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:28.165088 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 20 08:48:28.166102 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 20 08:48:28.166102 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:28.166102 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:28.166102 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 198/398] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:48:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:28.166102 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e64c2c77-99b4-462e-9e17-08101f725bea None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:28.245498 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:28.250453 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6abd0696-5455-4141-b580-e77767c3f79e None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:28.264883 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e64c2c77-99b4-462e-9e17-08101f725bea None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:28.267348 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:28.286565 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7088cec9-2c35-4f4d-af41-b93ec3bf65d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:28.287089 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7088cec9-2c35-4f4d-af41-b93ec3bf65d6 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:28.287363 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7088cec9-2c35-4f4d-af41-b93ec3bf65d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:28.287625 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7088cec9-2c35-4f4d-af41-b93ec3bf65d6 None admin] PATH_INFO: `/v3/domains` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:28.288254 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7088cec9-2c35-4f4d-af41-b93ec3bf65d6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:28.288548 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7088cec9-2c35-4f4d-af41-b93ec3bf65d6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:28.289907 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7088cec9-2c35-4f4d-af41-b93ec3bf65d6 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:28.298103 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20792a3c-9574-4e8d-b0e8-14832bb65876 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:28.298740 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20792a3c-9574-4e8d-b0e8-14832bb65876 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:28.299017 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20792a3c-9574-4e8d-b0e8-14832bb65876 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:28.299287 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20792a3c-9574-4e8d-b0e8-14832bb65876 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:28.300888 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 201/399] 10.222.0.36 () {56 vars in 1109 bytes} [Thu Oct 20 08:48:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:28.301722 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7088cec9-2c35-4f4d-af41-b93ec3bf65d6 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:28.311373 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20792a3c-9574-4e8d-b0e8-14832bb65876 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:28.312816 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20792a3c-9574-4e8d-b0e8-14832bb65876 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:28.324647 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-20792a3c-9574-4e8d-b0e8-14832bb65876 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Oct 20 08:48:28.324647 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:28.324647 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:28.324647 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:28.324647 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:28.324647 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:28.324647 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:28.324647 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:28.324647 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:28.324647 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:28.324647 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:28.324647 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:28.324647 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 20 08:48:28.324647 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 20 08:48:28.324647 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 20 08:48:28.324647 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 20 08:48:28.325668 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:28.325668 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:28.325668 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 20 08:48:28.325668 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 20 08:48:28.325668 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 20 08:48:28.325668 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 20 08:48:28.325668 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:28.325668 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:28.325668 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:28.325668 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:28.325668 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:28.325668 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:28.325668 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:28.325668 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:28.325668 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:28.325668 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:28.325668 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:28.325668 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:28.326949 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:28.326949 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:28.326949 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 20 08:48:28.326949 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 20 08:48:28.326949 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 20 08:48:28.326949 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 20 08:48:28.326949 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 20 08:48:28.326949 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 20 08:48:28.326949 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Oct 20 08:48:28.326949 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:28.326949 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 199/400] 10.222.0.36 () {56 vars in 1102 bytes} [Thu Oct 20 08:48:28 2022] GET /identity/v3/users/neutron => generated 85 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:28.326949 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20792a3c-9574-4e8d-b0e8-14832bb65876 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:28.405271 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7088cec9-2c35-4f4d-af41-b93ec3bf65d6 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:28.428733 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20792a3c-9574-4e8d-b0e8-14832bb65876 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:28.438242 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d841f6c2-b022-46d5-86d6-607123eebb97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:28.439192 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d841f6c2-b022-46d5-86d6-607123eebb97 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:28.439476 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d841f6c2-b022-46d5-86d6-607123eebb97 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:28.439746 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d841f6c2-b022-46d5-86d6-607123eebb97 None admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:28.440431 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d841f6c2-b022-46d5-86d6-607123eebb97 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:28.440735 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d841f6c2-b022-46d5-86d6-607123eebb97 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:28.442161 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d841f6c2-b022-46d5-86d6-607123eebb97 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:28.460413 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 202/401] 10.222.0.36 () {56 vars in 1148 bytes} [Thu Oct 20 08:48:28 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:28.461200 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d841f6c2-b022-46d5-86d6-607123eebb97 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:28.463050 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b17bab69-e922-4015-a526-3401fae89751 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:28.463904 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b17bab69-e922-4015-a526-3401fae89751 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:28.464978 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b17bab69-e922-4015-a526-3401fae89751 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:28.465382 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b17bab69-e922-4015-a526-3401fae89751 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:28.475712 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b17bab69-e922-4015-a526-3401fae89751 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:28.477170 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b17bab69-e922-4015-a526-3401fae89751 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:28.478895 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:28.491710 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-b17bab69-e922-4015-a526-3401fae89751 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:28.491710 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:28.491710 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:28.491710 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:28.491710 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:28.491710 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:28.491710 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:28.491710 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:28.491710 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:28.491710 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:28.491710 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:28.491710 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:28.491710 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 20 08:48:28.491710 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 20 08:48:28.491710 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 20 08:48:28.491710 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 20 08:48:28.492798 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:28.492798 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:28.492798 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:28.492798 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:28.492798 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:28.492798 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:28.492798 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:28.492798 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:28.492798 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:28.492798 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:28.492798 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:28.492798 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:28.492798 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:28.492798 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:28.492798 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:28.492798 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:28.492798 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 20 08:48:28.493853 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 20 08:48:28.493853 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:28.493853 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:28.493853 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 200/402] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:48:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:28.493853 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b17bab69-e922-4015-a526-3401fae89751 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:28.500496 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:28.566207 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d841f6c2-b022-46d5-86d6-607123eebb97 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:28.591545 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b17bab69-e922-4015-a526-3401fae89751 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:28.599559 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92ba8ef0-8379-4e12-9a56-49a492e92105 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:28.600158 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ba8ef0-8379-4e12-9a56-49a492e92105 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:28.600436 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ba8ef0-8379-4e12-9a56-49a492e92105 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:28.600701 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ba8ef0-8379-4e12-9a56-49a492e92105 None admin] PATH_INFO: `/v3/domains` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:28.601360 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92ba8ef0-8379-4e12-9a56-49a492e92105 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:28.601659 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92ba8ef0-8379-4e12-9a56-49a492e92105 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:28.603074 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92ba8ef0-8379-4e12-9a56-49a492e92105 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:28.613938 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 203/403] 10.222.0.36 () {56 vars in 1109 bytes} [Thu Oct 20 08:48:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:28.615430 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92ba8ef0-8379-4e12-9a56-49a492e92105 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:28.624484 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c036baa-6911-4717-8741-0ce8ccaaa156 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:28.625074 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c036baa-6911-4717-8741-0ce8ccaaa156 None admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:28.625355 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c036baa-6911-4717-8741-0ce8ccaaa156 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:28.625629 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c036baa-6911-4717-8741-0ce8ccaaa156 None admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:28.626257 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c036baa-6911-4717-8741-0ce8ccaaa156 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:28.627714 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c036baa-6911-4717-8741-0ce8ccaaa156 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:28.648190 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 201/404] 10.222.0.36 () {60 vars in 1132 bytes} [Thu Oct 20 08:48:28 2022] POST /identity/v3/users => generated 247 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 08:48:28.648963 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c036baa-6911-4717-8741-0ce8ccaaa156 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:28.708320 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:28.743001 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:28.751467 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92ba8ef0-8379-4e12-9a56-49a492e92105 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:28.764054 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c036baa-6911-4717-8741-0ce8ccaaa156 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:28.793156 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5225eb2-6dbf-4711-8b13-8b03e74893ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:28.793782 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5225eb2-6dbf-4711-8b13-8b03e74893ae None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:28.794061 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5225eb2-6dbf-4711-8b13-8b03e74893ae None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:28.794335 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5225eb2-6dbf-4711-8b13-8b03e74893ae None admin] PATH_INFO: `/v3/users` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:28.795118 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5225eb2-6dbf-4711-8b13-8b03e74893ae None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:28.795425 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5225eb2-6dbf-4711-8b13-8b03e74893ae None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:28.796899 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5225eb2-6dbf-4711-8b13-8b03e74893ae None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:28.803298 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fc90cef-46f0-4a31-9d07-fdd58077f270 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:28.804234 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc90cef-46f0-4a31-9d07-fdd58077f270 None admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:28.804675 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc90cef-46f0-4a31-9d07-fdd58077f270 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:28.805123 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc90cef-46f0-4a31-9d07-fdd58077f270 None admin] PATH_INFO: `/v3/projects/d9857b610fb542969cfb5efbbdb8ce61/users/a94e868115ec49918eda10724263930c/roles/f81100d13381464aa48cbae3561ca1bc` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:28.816857 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 204/405] 10.222.0.36 () {56 vars in 1139 bytes} [Thu Oct 20 08:48:28 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:28.817711 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5225eb2-6dbf-4711-8b13-8b03e74893ae None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:28.823814 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fc90cef-46f0-4a31-9d07-fdd58077f270 None admin] RBAC: Authorizing `identity:create_grant(project_id=d9857b610fb542969cfb5efbbdb8ce61, user_id=a94e868115ec49918eda10724263930c, role_id=f81100d13381464aa48cbae3561ca1bc)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:28.825310 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fc90cef-46f0-4a31-9d07-fdd58077f270 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:28.841634 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 202/406] 10.222.0.36 () {58 vars in 1439 bytes} [Thu Oct 20 08:48:28 2022] PUT /identity/v3/projects/d9857b610fb542969cfb5efbbdb8ce61/users/a94e868115ec49918eda10724263930c/roles/f81100d13381464aa48cbae3561ca1bc => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 08:48:28.842495 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fc90cef-46f0-4a31-9d07-fdd58077f270 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:28.941900 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5225eb2-6dbf-4711-8b13-8b03e74893ae None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:28.972847 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:28.976752 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fc90cef-46f0-4a31-9d07-fdd58077f270 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:29.012244 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7be6cdd3-4f0a-4cc7-8cd8-1bd316277079 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:29.013149 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be6cdd3-4f0a-4cc7-8cd8-1bd316277079 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:29.013433 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be6cdd3-4f0a-4cc7-8cd8-1bd316277079 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:29.013705 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be6cdd3-4f0a-4cc7-8cd8-1bd316277079 None admin] PATH_INFO: `/v3/domains` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:29.014369 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7be6cdd3-4f0a-4cc7-8cd8-1bd316277079 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:29.014713 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7be6cdd3-4f0a-4cc7-8cd8-1bd316277079 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:29.016076 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7be6cdd3-4f0a-4cc7-8cd8-1bd316277079 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:29.020218 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31ba4eba-c542-456f-beec-284139a86a3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:29.021185 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ba4eba-c542-456f-beec-284139a86a3c None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:29.021673 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ba4eba-c542-456f-beec-284139a86a3c None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:29.022131 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ba4eba-c542-456f-beec-284139a86a3c None admin] PATH_INFO: `/v3/projects/service` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:29.028469 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 205/407] 10.222.0.36 () {56 vars in 1109 bytes} [Thu Oct 20 08:48:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:29.029311 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7be6cdd3-4f0a-4cc7-8cd8-1bd316277079 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:29.043376 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31ba4eba-c542-456f-beec-284139a86a3c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:29.044861 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31ba4eba-c542-456f-beec-284139a86a3c None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:29.052814 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:29.060476 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-31ba4eba-c542-456f-beec-284139a86a3c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 20 08:48:29.060476 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:29.060476 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:29.060476 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:29.060476 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:29.060476 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:29.060476 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:29.060476 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:29.060476 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:29.060476 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:29.060476 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:29.060476 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:29.060476 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 20 08:48:29.060476 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 20 08:48:29.060476 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 20 08:48:29.061356 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 20 08:48:29.061356 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:29.061356 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:29.061356 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:29.061356 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:29.061356 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:29.061356 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:29.061356 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:29.061356 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:29.061356 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:29.061356 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:29.061356 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:29.061356 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:29.061356 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:29.061356 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:29.061356 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:29.061356 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:29.062227 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 20 08:48:29.062227 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 20 08:48:29.062227 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 20 08:48:29.062227 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 20 08:48:29.062227 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 20 08:48:29.062227 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 20 08:48:29.062227 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 20 08:48:29.062227 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:29.062227 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 203/408] 10.222.0.36 () {56 vars in 1111 bytes} [Thu Oct 20 08:48:28 2022] GET /identity/v3/projects/service => generated 88 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:29.062227 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31ba4eba-c542-456f-beec-284139a86a3c None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:29.146629 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7be6cdd3-4f0a-4cc7-8cd8-1bd316277079 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:29.182578 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78820deb-3783-4514-b0b2-ad6ceadb7f29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:29.183207 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78820deb-3783-4514-b0b2-ad6ceadb7f29 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:29.183483 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78820deb-3783-4514-b0b2-ad6ceadb7f29 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:29.183750 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78820deb-3783-4514-b0b2-ad6ceadb7f29 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:29.188257 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31ba4eba-c542-456f-beec-284139a86a3c None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:29.194834 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78820deb-3783-4514-b0b2-ad6ceadb7f29 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:29.196377 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78820deb-3783-4514-b0b2-ad6ceadb7f29 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:29.213866 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-78820deb-3783-4514-b0b2-ad6ceadb7f29 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:29.213866 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:29.213866 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:29.213866 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:29.213866 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:29.213866 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:29.213866 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:29.213866 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:29.213866 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:29.213866 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:29.213866 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:29.213866 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:29.213866 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 20 08:48:29.213866 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 20 08:48:29.213866 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 20 08:48:29.213866 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 20 08:48:29.214864 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:29.214864 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:29.214864 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:29.214864 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:29.214864 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:29.214864 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:29.214864 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:29.214864 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:29.214864 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:29.214864 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:29.214864 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:29.214864 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:29.214864 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:29.214864 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:29.214864 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:29.214864 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:29.214864 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 20 08:48:29.215702 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 20 08:48:29.215702 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:29.215702 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:48:29.219650 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 206/409] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:48:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:29.220485 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78820deb-3783-4514-b0b2-ad6ceadb7f29 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:29.227876 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e42c03db-be8f-4ade-8349-b39ac4926776 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:29.228504 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e42c03db-be8f-4ade-8349-b39ac4926776 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:29.228784 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e42c03db-be8f-4ade-8349-b39ac4926776 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:29.229055 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e42c03db-be8f-4ade-8349-b39ac4926776 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:29.238578 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e42c03db-be8f-4ade-8349-b39ac4926776 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:29.240929 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e42c03db-be8f-4ade-8349-b39ac4926776 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:29.258667 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-e42c03db-be8f-4ade-8349-b39ac4926776 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 20 08:48:29.258667 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:29.258667 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:29.258667 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:29.258667 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:29.258667 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:29.258667 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:29.258667 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:29.258667 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:29.258667 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:29.258667 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:29.258667 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:29.258667 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 20 08:48:29.258667 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 20 08:48:29.258667 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 20 08:48:29.259913 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 20 08:48:29.259913 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:29.259913 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:29.259913 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:29.259913 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:29.259913 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:29.259913 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:29.259913 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:29.259913 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:29.259913 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:29.259913 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:29.259913 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:29.259913 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:29.259913 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:29.259913 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:29.259913 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:29.259913 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:29.261689 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 20 08:48:29.261689 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 20 08:48:29.261689 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 20 08:48:29.261689 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 20 08:48:29.261689 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 20 08:48:29.261689 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 20 08:48:29.261689 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 20 08:48:29.261689 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:29.261689 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 204/410] 10.222.0.36 () {56 vars in 1111 bytes} [Thu Oct 20 08:48:29 2022] GET /identity/v3/projects/service => generated 88 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:29.262664 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e42c03db-be8f-4ade-8349-b39ac4926776 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:29.291275 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:29.294291 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:29.344899 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78820deb-3783-4514-b0b2-ad6ceadb7f29 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:29.380123 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e42c03db-be8f-4ade-8349-b39ac4926776 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:29.392317 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89725047-6675-4a7f-a1c9-4c80afa0700a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:29.393445 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89725047-6675-4a7f-a1c9-4c80afa0700a None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:29.393871 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89725047-6675-4a7f-a1c9-4c80afa0700a None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:29.394285 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89725047-6675-4a7f-a1c9-4c80afa0700a None admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:29.395398 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89725047-6675-4a7f-a1c9-4c80afa0700a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:29.396374 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89725047-6675-4a7f-a1c9-4c80afa0700a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:29.398308 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89725047-6675-4a7f-a1c9-4c80afa0700a None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:29.414965 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 207/411] 10.222.0.36 () {56 vars in 1148 bytes} [Thu Oct 20 08:48:29 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:29.416226 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89725047-6675-4a7f-a1c9-4c80afa0700a None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:29.417134 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79a04755-708e-4237-ab96-af6c11020c27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:29.417813 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79a04755-708e-4237-ab96-af6c11020c27 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:29.418189 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79a04755-708e-4237-ab96-af6c11020c27 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:29.418590 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79a04755-708e-4237-ab96-af6c11020c27 None admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:29.419666 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79a04755-708e-4237-ab96-af6c11020c27 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:29.420066 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79a04755-708e-4237-ab96-af6c11020c27 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:29.421797 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79a04755-708e-4237-ab96-af6c11020c27 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:29.434970 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 205/412] 10.222.0.36 () {56 vars in 1109 bytes} [Thu Oct 20 08:48:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:29.436133 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79a04755-708e-4237-ab96-af6c11020c27 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:29.527570 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:29.550944 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79a04755-708e-4237-ab96-af6c11020c27 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:29.555684 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89725047-6675-4a7f-a1c9-4c80afa0700a None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:29.589077 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-144d76ce-5877-40f2-8bdf-e5c5f9ca5f28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:29.589590 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:29.589984 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144d76ce-5877-40f2-8bdf-e5c5f9ca5f28 None admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:29.590432 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144d76ce-5877-40f2-8bdf-e5c5f9ca5f28 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:29.590769 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144d76ce-5877-40f2-8bdf-e5c5f9ca5f28 None admin] PATH_INFO: `/v3/projects/d9857b610fb542969cfb5efbbdb8ce61/users/ce043914cd9042909f70ab0adb272d34/roles/f81100d13381464aa48cbae3561ca1bc` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:29.600914 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6ec99af-34ae-4d80-9915-2636ab22abce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:29.601775 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ec99af-34ae-4d80-9915-2636ab22abce None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:29.602197 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ec99af-34ae-4d80-9915-2636ab22abce None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:29.602634 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ec99af-34ae-4d80-9915-2636ab22abce None admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:29.603467 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6ec99af-34ae-4d80-9915-2636ab22abce None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:29.603933 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6ec99af-34ae-4d80-9915-2636ab22abce None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:29.605523 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6ec99af-34ae-4d80-9915-2636ab22abce None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:29.608634 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-144d76ce-5877-40f2-8bdf-e5c5f9ca5f28 None admin] RBAC: Authorizing `identity:create_grant(project_id=d9857b610fb542969cfb5efbbdb8ce61, user_id=ce043914cd9042909f70ab0adb272d34, role_id=f81100d13381464aa48cbae3561ca1bc)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:29.610203 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-144d76ce-5877-40f2-8bdf-e5c5f9ca5f28 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:29.619319 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 208/413] 10.222.0.36 () {56 vars in 1148 bytes} [Thu Oct 20 08:48:29 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:29.620582 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6ec99af-34ae-4d80-9915-2636ab22abce None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:29.632818 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 206/414] 10.222.0.36 () {58 vars in 1439 bytes} [Thu Oct 20 08:48:29 2022] PUT /identity/v3/projects/d9857b610fb542969cfb5efbbdb8ce61/users/ce043914cd9042909f70ab0adb272d34/roles/f81100d13381464aa48cbae3561ca1bc => generated 0 bytes in 197 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 08:48:29.633803 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-144d76ce-5877-40f2-8bdf-e5c5f9ca5f28 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:29.759818 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-144d76ce-5877-40f2-8bdf-e5c5f9ca5f28 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:29.760459 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6ec99af-34ae-4d80-9915-2636ab22abce None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:29.822241 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:29.834395 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-021c207c-0c6d-4ed2-86ce-85f124c6e979 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:29.835423 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64a07c8e-4cf0-49cc-9758-eedad133c3ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:29.836341 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a07c8e-4cf0-49cc-9758-eedad133c3ad None admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:29.836820 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a07c8e-4cf0-49cc-9758-eedad133c3ad None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:29.837735 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021c207c-0c6d-4ed2-86ce-85f124c6e979 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:29.838237 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a07c8e-4cf0-49cc-9758-eedad133c3ad None admin] PATH_INFO: `/v3/projects/d9857b610fb542969cfb5efbbdb8ce61/users/7cfa6bb090de44eaa5bdf099a3096e8c/roles/f81100d13381464aa48cbae3561ca1bc` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:29.839457 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021c207c-0c6d-4ed2-86ce-85f124c6e979 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:29.839947 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021c207c-0c6d-4ed2-86ce-85f124c6e979 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:29.856805 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-021c207c-0c6d-4ed2-86ce-85f124c6e979 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:29.858327 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-021c207c-0c6d-4ed2-86ce-85f124c6e979 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:29.871351 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64a07c8e-4cf0-49cc-9758-eedad133c3ad None admin] RBAC: Authorizing `identity:create_grant(project_id=d9857b610fb542969cfb5efbbdb8ce61, user_id=7cfa6bb090de44eaa5bdf099a3096e8c, role_id=f81100d13381464aa48cbae3561ca1bc)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:29.872837 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64a07c8e-4cf0-49cc-9758-eedad133c3ad None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:29.881192 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-021c207c-0c6d-4ed2-86ce-85f124c6e979 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 20 08:48:29.881192 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:29.881192 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:29.881192 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:29.881192 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:29.881192 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:29.881192 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:29.881192 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:29.881192 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:29.881192 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:29.881192 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:29.881192 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:29.881192 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 20 08:48:29.881192 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 20 08:48:29.881192 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 20 08:48:29.882111 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 20 08:48:29.882111 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:29.882111 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:29.882111 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:29.882111 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:29.882111 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:29.882111 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:29.882111 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:29.882111 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:29.882111 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:29.882111 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:29.882111 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:29.882111 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:29.882111 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:29.882111 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:29.882111 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:29.882111 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:29.883475 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 20 08:48:29.883475 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 20 08:48:29.883475 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 20 08:48:29.883475 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 20 08:48:29.883475 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 20 08:48:29.883475 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 20 08:48:29.883475 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 20 08:48:29.883475 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:48:29.886939 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 209/415] 10.222.0.36 () {56 vars in 1111 bytes} [Thu Oct 20 08:48:29 2022] GET /identity/v3/projects/service => generated 88 bytes in 265 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:29.892343 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-021c207c-0c6d-4ed2-86ce-85f124c6e979 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:29.900940 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 207/416] 10.222.0.36 () {58 vars in 1439 bytes} [Thu Oct 20 08:48:29 2022] PUT /identity/v3/projects/d9857b610fb542969cfb5efbbdb8ce61/users/7cfa6bb090de44eaa5bdf099a3096e8c/roles/f81100d13381464aa48cbae3561ca1bc => generated 0 bytes in 267 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 08:48:29.948655 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:30.018977 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-021c207c-0c6d-4ed2-86ce-85f124c6e979 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:30.057706 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-468ff9fd-989e-42f7-9b93-a750e2951f97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:30.058742 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-468ff9fd-989e-42f7-9b93-a750e2951f97 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:30.059221 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-468ff9fd-989e-42f7-9b93-a750e2951f97 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:30.059711 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-468ff9fd-989e-42f7-9b93-a750e2951f97 None admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:30.060614 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-468ff9fd-989e-42f7-9b93-a750e2951f97 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:30.061127 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-468ff9fd-989e-42f7-9b93-a750e2951f97 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:30.062931 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-468ff9fd-989e-42f7-9b93-a750e2951f97 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:30.081781 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 210/417] 10.222.0.36 () {56 vars in 1148 bytes} [Thu Oct 20 08:48:29 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 08:48:30.084394 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64a07c8e-4cf0-49cc-9758-eedad133c3ad None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:30.204880 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64a07c8e-4cf0-49cc-9758-eedad133c3ad None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:30.247258 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce64e0d3-5314-4d2c-80b0-bf0cb51f9f3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:30.247921 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce64e0d3-5314-4d2c-80b0-bf0cb51f9f3f None admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:30.248201 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce64e0d3-5314-4d2c-80b0-bf0cb51f9f3f None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:30.248473 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce64e0d3-5314-4d2c-80b0-bf0cb51f9f3f None admin] PATH_INFO: `/v3/projects/d9857b610fb542969cfb5efbbdb8ce61/users/794ca389002d41748d7925216ed964a5/roles/f81100d13381464aa48cbae3561ca1bc` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:30.272756 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce64e0d3-5314-4d2c-80b0-bf0cb51f9f3f None admin] RBAC: Authorizing `identity:create_grant(project_id=d9857b610fb542969cfb5efbbdb8ce61, user_id=794ca389002d41748d7925216ed964a5, role_id=f81100d13381464aa48cbae3561ca1bc)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:30.274146 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce64e0d3-5314-4d2c-80b0-bf0cb51f9f3f None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:30.300267 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 208/418] 10.222.0.36 () {58 vars in 1439 bytes} [Thu Oct 20 08:48:30 2022] PUT /identity/v3/projects/d9857b610fb542969cfb5efbbdb8ce61/users/794ca389002d41748d7925216ed964a5/roles/f81100d13381464aa48cbae3561ca1bc => generated 0 bytes in 217 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 08:48:31.118169 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e1f05d-4397-4cbc-b4dc-994aaf03e7be None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:31.119204 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e1f05d-4397-4cbc-b4dc-994aaf03e7be None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:31.119700 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e1f05d-4397-4cbc-b4dc-994aaf03e7be None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:31.121525 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 211/419] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:48:31 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:48:31.126432 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49059ba-5d60-4c84-a010-e0a16561a652 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:31.126749 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49059ba-5d60-4c84-a010-e0a16561a652 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:31.127013 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49059ba-5d60-4c84-a010-e0a16561a652 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:31.152175 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-e49059ba-5d60-4c84-a010-e0a16561a652 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:48:31.153367 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-e49059ba-5d60-4c84-a010-e0a16561a652 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:48:31.187499 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:31.244751 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 209/420] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:48:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 20 08:48:31.551501 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42abc86d-01e3-46c0-8bc0-d4afb800a20d None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:31.551817 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42abc86d-01e3-46c0-8bc0-d4afb800a20d None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:31.552049 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42abc86d-01e3-46c0-8bc0-d4afb800a20d None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:31.554203 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 212/421] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:48:31 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:48:31.561414 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63329fc7-702e-402b-a504-0eabf396b2a5 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:31.561706 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63329fc7-702e-402b-a504-0eabf396b2a5 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:31.561979 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63329fc7-702e-402b-a504-0eabf396b2a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:31.591740 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-63329fc7-702e-402b-a504-0eabf396b2a5 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:48:31.592964 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-63329fc7-702e-402b-a504-0eabf396b2a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:48:31.673516 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42abc86d-01e3-46c0-8bc0-d4afb800a20d None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:31.690399 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 210/422] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:48:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 20 08:48:31.807192 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42abc86d-01e3-46c0-8bc0-d4afb800a20d None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:31.815437 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:31.856055 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b588395-ead5-4b36-b2f9-9f5579b9fccd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:31.856689 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b588395-ead5-4b36-b2f9-9f5579b9fccd None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:31.856971 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b588395-ead5-4b36-b2f9-9f5579b9fccd None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:31.857238 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b588395-ead5-4b36-b2f9-9f5579b9fccd None admin] PATH_INFO: `/v3/roles/service` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:31.870071 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b588395-ead5-4b36-b2f9-9f5579b9fccd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 20 08:48:31.874830 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-7b588395-ead5-4b36-b2f9-9f5579b9fccd None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 20 08:48:31.874830 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:31.874830 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:31.874830 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:31.874830 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:31.874830 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:31.874830 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:31.874830 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:31.874830 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:31.874830 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:31.874830 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:31.874830 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:31.874830 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 20 08:48:31.874830 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 20 08:48:31.874830 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 20 08:48:31.874830 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 20 08:48:31.875698 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 20 08:48:31.875698 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 20 08:48:31.875698 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 20 08:48:31.875698 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 20 08:48:31.875698 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:31.875698 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:31.875698 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:31.875698 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:31.875698 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:31.875698 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:31.875698 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:31.875698 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:31.875698 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:31.875698 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:31.875698 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:31.875698 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:31.876921 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:31.876921 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:31.876921 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:31.876921 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:31.876921 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 20 08:48:31.876921 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 20 08:48:31.876921 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 20 08:48:31.876921 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 20 08:48:31.876921 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 20 08:48:31.876921 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 20 08:48:31.876921 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 20 08:48:31.876921 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:48:31.876921 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 213/423] 10.222.0.36 () {56 vars in 1102 bytes} [Thu Oct 20 08:48:31 2022] GET /identity/v3/roles/service => generated 85 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:31.882524 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63329fc7-702e-402b-a504-0eabf396b2a5 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:32.074922 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:32.085466 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63329fc7-702e-402b-a504-0eabf396b2a5 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:32.134220 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f2e1f1e-bca1-4a3e-baa3-aa6f06f896aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:32.134854 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2e1f1e-bca1-4a3e-baa3-aa6f06f896aa None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:32.135128 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2e1f1e-bca1-4a3e-baa3-aa6f06f896aa None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:32.135395 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2e1f1e-bca1-4a3e-baa3-aa6f06f896aa None admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:32.135579 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b588395-ead5-4b36-b2f9-9f5579b9fccd None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:32.136041 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f2e1f1e-bca1-4a3e-baa3-aa6f06f896aa None admin] RBAC: Adding query filter params (name=service) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:32.136343 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f2e1f1e-bca1-4a3e-baa3-aa6f06f896aa None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:32.137715 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f2e1f1e-bca1-4a3e-baa3-aa6f06f896aa None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:32.146669 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 211/424] 10.222.0.36 () {56 vars in 1103 bytes} [Thu Oct 20 08:48:31 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 264 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:32.156300 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f2e1f1e-bca1-4a3e-baa3-aa6f06f896aa None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:32.276276 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f2e1f1e-bca1-4a3e-baa3-aa6f06f896aa None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:32.314001 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5718733e-6a4c-4439-9e62-dac790eddee7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:32.314658 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5718733e-6a4c-4439-9e62-dac790eddee7 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:32.314934 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5718733e-6a4c-4439-9e62-dac790eddee7 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:32.315198 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5718733e-6a4c-4439-9e62-dac790eddee7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:32.327671 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5718733e-6a4c-4439-9e62-dac790eddee7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:32.329285 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5718733e-6a4c-4439-9e62-dac790eddee7 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:32.334250 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b588395-ead5-4b36-b2f9-9f5579b9fccd None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:32.345571 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-5718733e-6a4c-4439-9e62-dac790eddee7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:32.345571 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:32.345571 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:32.345571 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:32.345571 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:32.345571 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:32.345571 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:32.345571 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:32.345571 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:32.345571 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:32.345571 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:32.345571 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:32.345571 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 20 08:48:32.345571 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 20 08:48:32.345571 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 20 08:48:32.345571 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 20 08:48:32.346560 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:32.346560 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:32.346560 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:32.346560 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:32.346560 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:32.346560 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:32.346560 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:32.346560 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:32.346560 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:32.346560 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:32.346560 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:32.346560 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:32.346560 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:32.346560 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:32.346560 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:32.346560 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:32.346560 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 20 08:48:32.347421 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 20 08:48:32.347421 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:32.347421 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:32.347421 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 212/425] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:48:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 20 08:48:32.354061 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5a6c72-a8b1-4135-8ef5-78fe8c954955 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:32.354061 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5a6c72-a8b1-4135-8ef5-78fe8c954955 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:32.354061 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5a6c72-a8b1-4135-8ef5-78fe8c954955 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:32.354061 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 213/426] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:48:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:48:32.354061 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c5a6c72-a8b1-4135-8ef5-78fe8c954955 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:32.358570 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:32.388585 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-073eced2-03e6-4407-83cf-5773a00a5483 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:32.389617 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073eced2-03e6-4407-83cf-5773a00a5483 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:32.390077 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073eced2-03e6-4407-83cf-5773a00a5483 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:32.390560 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073eced2-03e6-4407-83cf-5773a00a5483 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:32.404811 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-073eced2-03e6-4407-83cf-5773a00a5483 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 20 08:48:32.410589 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-073eced2-03e6-4407-83cf-5773a00a5483 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 20 08:48:32.410589 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:32.410589 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:32.410589 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:32.410589 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:32.410589 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:32.410589 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:32.410589 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:32.410589 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:32.410589 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:32.410589 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:32.410589 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:32.410589 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 20 08:48:32.410589 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 20 08:48:32.410589 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 20 08:48:32.410589 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 20 08:48:32.412222 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 20 08:48:32.412222 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 20 08:48:32.412222 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 20 08:48:32.412222 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 20 08:48:32.412222 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:32.412222 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:32.412222 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:32.412222 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:32.412222 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:32.412222 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:32.412222 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:32.412222 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:32.412222 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:32.412222 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:32.412222 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:32.412222 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:32.413018 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:32.413018 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:32.413018 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:32.413018 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:32.413018 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 20 08:48:32.413018 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 20 08:48:32.413018 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 20 08:48:32.413018 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 20 08:48:32.413018 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 20 08:48:32.413018 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 20 08:48:32.413018 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 20 08:48:32.413018 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:48:32.415938 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 214/427] 10.222.0.36 () {56 vars in 1102 bytes} [Thu Oct 20 08:48:32 2022] GET /identity/v3/roles/service => generated 85 bytes in 283 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 20 08:48:32.417612 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12bebd61-b8b3-45b6-891c-df64e0027c05 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:32.418134 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12bebd61-b8b3-45b6-891c-df64e0027c05 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:32.418626 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12bebd61-b8b3-45b6-891c-df64e0027c05 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:32.438877 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-12bebd61-b8b3-45b6-891c-df64e0027c05 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:48:32.440464 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-12bebd61-b8b3-45b6-891c-df64e0027c05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:48:32.464812 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:32.477836 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c5a6c72-a8b1-4135-8ef5-78fe8c954955 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:32.514131 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62825b94-282a-447c-a3e5-34049e03441b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:32.514775 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62825b94-282a-447c-a3e5-34049e03441b None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:32.515202 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 215/428] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:48:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 20 08:48:32.515310 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62825b94-282a-447c-a3e5-34049e03441b None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:32.515310 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62825b94-282a-447c-a3e5-34049e03441b None admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:32.515925 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62825b94-282a-447c-a3e5-34049e03441b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:32.516272 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62825b94-282a-447c-a3e5-34049e03441b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:32.517630 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62825b94-282a-447c-a3e5-34049e03441b None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:32.519696 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12bebd61-b8b3-45b6-891c-df64e0027c05 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:32.531806 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 214/429] 10.222.0.36 () {56 vars in 1109 bytes} [Thu Oct 20 08:48:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:32.532794 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b90152-0146-42e3-9fdb-bac61769df14 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:32.533142 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b90152-0146-42e3-9fdb-bac61769df14 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:32.533497 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b90152-0146-42e3-9fdb-bac61769df14 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:32.534841 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 215/430] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:48:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:48:32.540425 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26b90152-0146-42e3-9fdb-bac61769df14 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:32.607850 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:32.652892 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12bebd61-b8b3-45b6-891c-df64e0027c05 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:32.661241 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26b90152-0146-42e3-9fdb-bac61769df14 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:32.688539 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f0d5831-0e3c-4fe9-95f6-aedacc7fb1d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:32.689790 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f0d5831-0e3c-4fe9-95f6-aedacc7fb1d6 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:32.690219 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f0d5831-0e3c-4fe9-95f6-aedacc7fb1d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:32.690936 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f0d5831-0e3c-4fe9-95f6-aedacc7fb1d6 None admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:32.691763 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f0d5831-0e3c-4fe9-95f6-aedacc7fb1d6 None admin] RBAC: Adding query filter params (name=service) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:32.692371 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f0d5831-0e3c-4fe9-95f6-aedacc7fb1d6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:32.695099 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f0d5831-0e3c-4fe9-95f6-aedacc7fb1d6 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:32.704276 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de77a15c-3dee-49e7-b3aa-d3be548f2f3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:32.705041 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de77a15c-3dee-49e7-b3aa-d3be548f2f3d None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:32.705412 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de77a15c-3dee-49e7-b3aa-d3be548f2f3d None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:32.705821 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de77a15c-3dee-49e7-b3aa-d3be548f2f3d None admin] PATH_INFO: `/v3/users/swift` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:32.707052 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 216/431] 10.222.0.36 () {56 vars in 1103 bytes} [Thu Oct 20 08:48:32 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:32.709190 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af0d5be-f7fa-4f46-a1cd-119bcbb76725 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:32.709605 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af0d5be-f7fa-4f46-a1cd-119bcbb76725 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:32.710004 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af0d5be-f7fa-4f46-a1cd-119bcbb76725 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:32.719189 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de77a15c-3dee-49e7-b3aa-d3be548f2f3d None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:32.720655 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de77a15c-3dee-49e7-b3aa-d3be548f2f3d None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:32.731036 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-4af0d5be-f7fa-4f46-a1cd-119bcbb76725 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:48:32.732440 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-4af0d5be-f7fa-4f46-a1cd-119bcbb76725 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:48:32.732602 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-de77a15c-3dee-49e7-b3aa-d3be548f2f3d None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 20 08:48:32.732602 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:32.732602 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:32.732602 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:32.732602 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:32.732602 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:32.732602 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:32.732602 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:32.732602 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:32.732602 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:32.732602 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:32.732602 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:32.732602 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 20 08:48:32.732602 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 20 08:48:32.732602 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 20 08:48:32.732602 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 20 08:48:32.733489 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:32.733489 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:32.733489 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 20 08:48:32.733489 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 20 08:48:32.733489 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 20 08:48:32.733489 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 20 08:48:32.733489 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:32.733489 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:32.733489 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:32.733489 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:32.733489 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:32.733489 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:32.733489 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:32.733489 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:32.733489 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:32.733489 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:32.733489 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:32.733489 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:32.734395 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:32.734395 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:32.734395 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 20 08:48:32.734395 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 20 08:48:32.734395 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 20 08:48:32.734395 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 20 08:48:32.734395 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 20 08:48:32.734395 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 20 08:48:32.734395 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 20 08:48:32.734395 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:32.738687 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 216/432] 10.222.0.36 () {56 vars in 1096 bytes} [Thu Oct 20 08:48:32 2022] GET /identity/v3/users/swift => generated 83 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:32.739714 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de77a15c-3dee-49e7-b3aa-d3be548f2f3d None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:32.753209 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:32.825825 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 217/433] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:48:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 20 08:48:32.828141 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af0d5be-f7fa-4f46-a1cd-119bcbb76725 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:32.852797 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de77a15c-3dee-49e7-b3aa-d3be548f2f3d None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:32.885843 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:32.900374 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bebab071-43be-4fe9-90a6-0a21c76a3889 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:32.901011 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bebab071-43be-4fe9-90a6-0a21c76a3889 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:32.901311 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bebab071-43be-4fe9-90a6-0a21c76a3889 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:32.901923 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bebab071-43be-4fe9-90a6-0a21c76a3889 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:32.914960 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bebab071-43be-4fe9-90a6-0a21c76a3889 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:32.916377 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bebab071-43be-4fe9-90a6-0a21c76a3889 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:32.933351 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-bebab071-43be-4fe9-90a6-0a21c76a3889 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:32.933351 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:32.933351 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:32.933351 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:32.933351 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:32.933351 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:32.933351 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:32.933351 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:32.933351 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:32.933351 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:32.933351 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:32.933351 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:32.933351 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 20 08:48:32.933351 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 20 08:48:32.933351 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 20 08:48:32.933351 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 20 08:48:32.934295 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:32.934295 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:32.934295 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:32.934295 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:32.934295 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:32.934295 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:32.934295 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:32.934295 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:32.934295 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:32.934295 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:32.934295 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:32.934295 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:32.934295 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:32.934295 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:32.934295 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:32.934295 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:32.934295 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 20 08:48:32.935181 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 20 08:48:32.935181 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:32.935181 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:32.935181 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 217/434] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:48:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:32.935889 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bebab071-43be-4fe9-90a6-0a21c76a3889 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:32.975231 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af0d5be-f7fa-4f46-a1cd-119bcbb76725 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:33.010818 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe8be0bf-80a9-427c-9daa-2d9cc7f53489 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:33.011419 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe8be0bf-80a9-427c-9daa-2d9cc7f53489 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:33.011683 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe8be0bf-80a9-427c-9daa-2d9cc7f53489 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:33.011954 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe8be0bf-80a9-427c-9daa-2d9cc7f53489 None admin] PATH_INFO: `/v3/users` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:33.012665 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe8be0bf-80a9-427c-9daa-2d9cc7f53489 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:33.012964 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe8be0bf-80a9-427c-9daa-2d9cc7f53489 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:33.014453 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe8be0bf-80a9-427c-9daa-2d9cc7f53489 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:33.032891 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 218/435] 10.222.0.36 () {56 vars in 1135 bytes} [Thu Oct 20 08:48:32 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:33.033837 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe8be0bf-80a9-427c-9daa-2d9cc7f53489 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:33.042533 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:33.056671 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bebab071-43be-4fe9-90a6-0a21c76a3889 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:33.092866 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b06eaccc-e7ee-425d-a3f8-335cc89f4b1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:33.093481 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b06eaccc-e7ee-425d-a3f8-335cc89f4b1b None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:33.093755 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b06eaccc-e7ee-425d-a3f8-335cc89f4b1b None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:33.094017 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b06eaccc-e7ee-425d-a3f8-335cc89f4b1b None admin] PATH_INFO: `/v3/roles/service` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:33.104136 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b06eaccc-e7ee-425d-a3f8-335cc89f4b1b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 20 08:48:33.109280 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-b06eaccc-e7ee-425d-a3f8-335cc89f4b1b None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 20 08:48:33.109280 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:33.109280 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:33.109280 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:33.109280 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:33.109280 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:33.109280 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:33.109280 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:33.109280 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:33.109280 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:33.109280 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:33.109280 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:33.109280 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 20 08:48:33.109280 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 20 08:48:33.109280 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 20 08:48:33.109280 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 20 08:48:33.110303 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 20 08:48:33.110303 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 20 08:48:33.110303 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 20 08:48:33.110303 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 20 08:48:33.110303 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:33.110303 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:33.110303 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:33.110303 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:33.110303 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:33.110303 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:33.110303 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:33.110303 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:33.110303 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:33.110303 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:33.110303 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:33.110303 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:33.111297 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:33.111297 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:33.111297 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:33.111297 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:33.111297 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 20 08:48:33.111297 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 20 08:48:33.111297 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 20 08:48:33.111297 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 20 08:48:33.111297 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 20 08:48:33.111297 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 20 08:48:33.111297 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 20 08:48:33.111297 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:33.111297 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 218/436] 10.222.0.36 () {56 vars in 1102 bytes} [Thu Oct 20 08:48:32 2022] GET /identity/v3/roles/service => generated 85 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:33.112092 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c51eac59-1b69-483d-8506-a715036a484c None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:33.112092 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c51eac59-1b69-483d-8506-a715036a484c None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:33.112237 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c51eac59-1b69-483d-8506-a715036a484c None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:33.113340 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 219/437] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:48:33 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:48:33.114016 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c51eac59-1b69-483d-8506-a715036a484c None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:33.148111 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe8be0bf-80a9-427c-9daa-2d9cc7f53489 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:33.151672 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:33.181127 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f35272b-30e7-4e1b-8098-d697a0a46131 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:33.181725 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f35272b-30e7-4e1b-8098-d697a0a46131 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:33.181998 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f35272b-30e7-4e1b-8098-d697a0a46131 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:33.182264 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f35272b-30e7-4e1b-8098-d697a0a46131 None admin] PATH_INFO: `/v3/domains` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:33.182927 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f35272b-30e7-4e1b-8098-d697a0a46131 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:33.183219 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f35272b-30e7-4e1b-8098-d697a0a46131 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:33.184604 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f35272b-30e7-4e1b-8098-d697a0a46131 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:33.199654 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 219/438] 10.222.0.36 () {56 vars in 1109 bytes} [Thu Oct 20 08:48:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:33.200427 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f35272b-30e7-4e1b-8098-d697a0a46131 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:33.217899 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c51eac59-1b69-483d-8506-a715036a484c None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:33.250546 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d69bdd8b-78a0-4f2d-b253-a98cc128c7bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:33.251159 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69bdd8b-78a0-4f2d-b253-a98cc128c7bf None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:33.251441 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69bdd8b-78a0-4f2d-b253-a98cc128c7bf None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:33.251715 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69bdd8b-78a0-4f2d-b253-a98cc128c7bf None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:33.263103 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d69bdd8b-78a0-4f2d-b253-a98cc128c7bf None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:33.263945 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:33.264765 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d69bdd8b-78a0-4f2d-b253-a98cc128c7bf None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:33.279070 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-d69bdd8b-78a0-4f2d-b253-a98cc128c7bf None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:33.279070 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:33.279070 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:33.279070 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:33.279070 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:33.279070 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:33.279070 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:33.279070 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:33.279070 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:33.279070 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:33.279070 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:33.279070 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:33.279070 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 20 08:48:33.279070 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 20 08:48:33.279070 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 20 08:48:33.279070 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 20 08:48:33.280417 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:33.280417 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:33.280417 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:33.280417 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:33.280417 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:33.280417 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:33.280417 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:33.280417 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:33.280417 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:33.280417 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:33.280417 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:33.280417 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:33.280417 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:33.280417 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:33.280417 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:33.280417 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:33.280417 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 20 08:48:33.281283 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 20 08:48:33.281283 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:33.281283 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:33.281283 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 220/439] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:48:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:33.281283 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d69bdd8b-78a0-4f2d-b253-a98cc128c7bf None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:33.300914 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f35272b-30e7-4e1b-8098-d697a0a46131 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:33.334336 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a80ba61e-f45d-4424-8eb1-ae53c961896c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:33.334993 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a80ba61e-f45d-4424-8eb1-ae53c961896c None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:33.335271 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a80ba61e-f45d-4424-8eb1-ae53c961896c None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:33.335539 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a80ba61e-f45d-4424-8eb1-ae53c961896c None admin] PATH_INFO: `/v3/roles/service` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:33.345446 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a80ba61e-f45d-4424-8eb1-ae53c961896c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 20 08:48:33.351162 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-a80ba61e-f45d-4424-8eb1-ae53c961896c None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 20 08:48:33.351162 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:33.351162 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:33.351162 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:33.351162 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:33.351162 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:33.351162 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:33.351162 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:33.351162 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:33.351162 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:33.351162 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:33.351162 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:33.351162 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 20 08:48:33.351162 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 20 08:48:33.351162 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 20 08:48:33.351162 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 20 08:48:33.352081 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 20 08:48:33.352081 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 20 08:48:33.352081 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 20 08:48:33.352081 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 20 08:48:33.352081 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:33.352081 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:33.352081 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:33.352081 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:33.352081 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:33.352081 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:33.352081 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:33.352081 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:33.352081 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:33.352081 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:33.352081 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:33.352081 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:33.352878 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:33.352878 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:33.352878 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:33.352878 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:33.352878 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 20 08:48:33.352878 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 20 08:48:33.352878 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 20 08:48:33.352878 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 20 08:48:33.352878 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 20 08:48:33.352878 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 20 08:48:33.352878 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 20 08:48:33.352878 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:48:33.352878 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 220/440] 10.222.0.36 () {56 vars in 1102 bytes} [Thu Oct 20 08:48:33 2022] GET /identity/v3/roles/service => generated 85 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:33.353692 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-344affe2-e944-47ab-8a02-6732295bb38a None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:33.353922 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-344affe2-e944-47ab-8a02-6732295bb38a None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:33.354194 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-344affe2-e944-47ab-8a02-6732295bb38a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:33.371644 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-344affe2-e944-47ab-8a02-6732295bb38a None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:48:33.372193 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:33.373246 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-344affe2-e944-47ab-8a02-6732295bb38a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:48:33.384955 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d69bdd8b-78a0-4f2d-b253-a98cc128c7bf None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:33.418839 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9a0608a-2613-45c6-9e26-a66369d9f3e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:33.419370 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a0608a-2613-45c6-9e26-a66369d9f3e5 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:33.419792 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a0608a-2613-45c6-9e26-a66369d9f3e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:33.419912 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a0608a-2613-45c6-9e26-a66369d9f3e5 None admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:33.420555 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9a0608a-2613-45c6-9e26-a66369d9f3e5 None admin] RBAC: Adding query filter params (name=service) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:33.420855 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9a0608a-2613-45c6-9e26-a66369d9f3e5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:33.422347 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9a0608a-2613-45c6-9e26-a66369d9f3e5 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:33.430136 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 221/441] 10.222.0.36 () {56 vars in 1103 bytes} [Thu Oct 20 08:48:33 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:33.431007 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9a0608a-2613-45c6-9e26-a66369d9f3e5 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:33.439790 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 221/442] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:48:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 20 08:48:33.440768 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-344affe2-e944-47ab-8a02-6732295bb38a None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:33.497838 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:33.539405 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9a0608a-2613-45c6-9e26-a66369d9f3e5 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:33.543577 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-344affe2-e944-47ab-8a02-6732295bb38a None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:33.572702 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1eb093-05b8-4915-8829-a6215deed354 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:33.573314 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1eb093-05b8-4915-8829-a6215deed354 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:33.573593 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1eb093-05b8-4915-8829-a6215deed354 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:33.573868 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1eb093-05b8-4915-8829-a6215deed354 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:33.581176 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed1eb093-05b8-4915-8829-a6215deed354 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:33.582322 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6412b972-0692-4501-82da-253463384718 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:33.582645 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed1eb093-05b8-4915-8829-a6215deed354 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:33.584189 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6412b972-0692-4501-82da-253463384718 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:33.584702 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6412b972-0692-4501-82da-253463384718 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:33.585300 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6412b972-0692-4501-82da-253463384718 None admin] PATH_INFO: `/v3/domains` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:33.586595 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6412b972-0692-4501-82da-253463384718 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:33.587068 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6412b972-0692-4501-82da-253463384718 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:33.588627 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6412b972-0692-4501-82da-253463384718 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:33.594318 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-ed1eb093-05b8-4915-8829-a6215deed354 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 20 08:48:33.594318 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:33.594318 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:33.594318 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:33.594318 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:33.594318 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:33.594318 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:33.594318 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:33.594318 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:33.594318 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:33.594318 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:33.594318 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:33.594318 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 20 08:48:33.594318 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 20 08:48:33.594318 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 20 08:48:33.594318 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 20 08:48:33.595317 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:33.595317 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:33.595317 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 20 08:48:33.595317 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 20 08:48:33.595317 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 20 08:48:33.595317 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 20 08:48:33.595317 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:33.595317 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:33.595317 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:33.595317 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:33.595317 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:33.595317 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:33.595317 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:33.595317 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:33.595317 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:33.595317 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:33.595317 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:33.595317 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:33.596237 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:33.596237 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:33.596237 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 20 08:48:33.596237 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 20 08:48:33.596237 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 20 08:48:33.596237 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 20 08:48:33.596237 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 20 08:48:33.596237 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 20 08:48:33.596237 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 20 08:48:33.596237 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:33.600119 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 222/443] 10.222.0.36 () {56 vars in 1093 bytes} [Thu Oct 20 08:48:33 2022] GET /identity/v3/users/nova => generated 82 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:33.600750 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1eb093-05b8-4915-8829-a6215deed354 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:33.605713 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 222/444] 10.222.0.36 () {56 vars in 1109 bytes} [Thu Oct 20 08:48:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:33.605713 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6412b972-0692-4501-82da-253463384718 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:33.615517 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:33.713874 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1eb093-05b8-4915-8829-a6215deed354 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:33.723638 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6412b972-0692-4501-82da-253463384718 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:33.739308 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:33.746964 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38da82e8-725a-4f9b-a024-7dde0efc25f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:33.747551 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38da82e8-725a-4f9b-a024-7dde0efc25f3 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:33.747824 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38da82e8-725a-4f9b-a024-7dde0efc25f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:33.748086 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38da82e8-725a-4f9b-a024-7dde0efc25f3 None admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:33.748725 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38da82e8-725a-4f9b-a024-7dde0efc25f3 None admin] RBAC: Adding query filter params (name=service) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:33.749023 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38da82e8-725a-4f9b-a024-7dde0efc25f3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:33.750373 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38da82e8-725a-4f9b-a024-7dde0efc25f3 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:33.757860 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3a09370-4814-4802-ad1e-03390a66d4cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:33.758496 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a09370-4814-4802-ad1e-03390a66d4cc None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:33.758794 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a09370-4814-4802-ad1e-03390a66d4cc None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:33.758977 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 223/445] 10.222.0.36 () {56 vars in 1103 bytes} [Thu Oct 20 08:48:33 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:33.759091 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a09370-4814-4802-ad1e-03390a66d4cc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:33.759802 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38da82e8-725a-4f9b-a024-7dde0efc25f3 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:33.770208 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3a09370-4814-4802-ad1e-03390a66d4cc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:33.772245 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3a09370-4814-4802-ad1e-03390a66d4cc None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:33.786786 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-a3a09370-4814-4802-ad1e-03390a66d4cc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:33.786786 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:33.786786 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:33.786786 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:33.786786 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:33.786786 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:33.786786 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:33.786786 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:33.786786 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:33.786786 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:33.786786 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:33.786786 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:33.786786 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 20 08:48:33.786786 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 20 08:48:33.786786 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 20 08:48:33.786786 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 20 08:48:33.787751 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:33.787751 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:33.787751 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:33.787751 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:33.787751 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:33.787751 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:33.787751 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:33.787751 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:33.787751 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:33.787751 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:33.787751 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:33.787751 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:33.787751 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:33.787751 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:33.787751 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:33.787751 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:33.787751 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 20 08:48:33.788740 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 20 08:48:33.788740 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:33.788740 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:48:33.788740 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 223/446] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:48:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:33.788740 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3a09370-4814-4802-ad1e-03390a66d4cc None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:33.830284 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:33.861039 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38da82e8-725a-4f9b-a024-7dde0efc25f3 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:33.886989 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3a09370-4814-4802-ad1e-03390a66d4cc None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:33.894196 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a8a89bb-0507-4dc3-be9c-7986608a13f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:33.894824 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8a89bb-0507-4dc3-be9c-7986608a13f8 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:33.895102 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8a89bb-0507-4dc3-be9c-7986608a13f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:33.895372 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8a89bb-0507-4dc3-be9c-7986608a13f8 None admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:33.896133 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a8a89bb-0507-4dc3-be9c-7986608a13f8 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:33.896429 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a8a89bb-0507-4dc3-be9c-7986608a13f8 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:33.897858 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a8a89bb-0507-4dc3-be9c-7986608a13f8 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:33.921524 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-234fdebf-b060-41e4-a060-8bbd67b92b53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:33.922148 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234fdebf-b060-41e4-a060-8bbd67b92b53 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:33.922428 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234fdebf-b060-41e4-a060-8bbd67b92b53 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:33.922739 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234fdebf-b060-41e4-a060-8bbd67b92b53 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:33.929930 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-234fdebf-b060-41e4-a060-8bbd67b92b53 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:33.931430 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-234fdebf-b060-41e4-a060-8bbd67b92b53 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:33.934033 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 224/447] 10.222.0.36 () {56 vars in 1133 bytes} [Thu Oct 20 08:48:33 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:33.934885 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a8a89bb-0507-4dc3-be9c-7986608a13f8 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:33.943898 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-234fdebf-b060-41e4-a060-8bbd67b92b53 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 20 08:48:33.943898 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:33.943898 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:33.943898 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:33.943898 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:33.943898 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:33.943898 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:33.943898 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:33.943898 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:33.943898 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:33.943898 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:33.943898 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:33.943898 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 20 08:48:33.943898 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 20 08:48:33.943898 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 20 08:48:33.944822 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 20 08:48:33.944822 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:33.944822 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:33.944822 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:33.944822 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:33.944822 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:33.944822 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:33.944822 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:33.944822 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:33.944822 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:33.944822 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:33.944822 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:33.944822 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:33.944822 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:33.944822 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:33.944822 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:33.944822 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:33.945689 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 20 08:48:33.945689 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 20 08:48:33.945689 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 20 08:48:33.945689 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 20 08:48:33.945689 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 20 08:48:33.945689 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 20 08:48:33.945689 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 20 08:48:33.945689 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:48:33.945689 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 224/448] 10.222.0.36 () {56 vars in 1111 bytes} [Thu Oct 20 08:48:33 2022] GET /identity/v3/projects/service => generated 88 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:33.946679 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-234fdebf-b060-41e4-a060-8bbd67b92b53 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:33.976331 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:34.037950 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a8a89bb-0507-4dc3-be9c-7986608a13f8 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:34.047119 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-234fdebf-b060-41e4-a060-8bbd67b92b53 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:34.065661 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:34.072830 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5af3476c-9969-48e0-b270-e5d6520001e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:34.073452 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af3476c-9969-48e0-b270-e5d6520001e1 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:34.073725 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af3476c-9969-48e0-b270-e5d6520001e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:34.073998 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af3476c-9969-48e0-b270-e5d6520001e1 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:34.080505 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebd695b5-3e98-4878-95bb-833125a8e0ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:34.081117 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd695b5-3e98-4878-95bb-833125a8e0ab None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:34.081397 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd695b5-3e98-4878-95bb-833125a8e0ab None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:34.081663 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd695b5-3e98-4878-95bb-833125a8e0ab None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:34.084202 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5af3476c-9969-48e0-b270-e5d6520001e1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 20 08:48:34.088900 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-5af3476c-9969-48e0-b270-e5d6520001e1 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 20 08:48:34.088900 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:34.088900 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:34.088900 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:34.088900 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:34.088900 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:34.088900 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:34.088900 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:34.088900 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:34.088900 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:34.088900 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:34.088900 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:34.088900 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 20 08:48:34.088900 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 20 08:48:34.088900 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 20 08:48:34.088900 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 20 08:48:34.089829 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 20 08:48:34.089829 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 20 08:48:34.089829 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 20 08:48:34.089829 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 20 08:48:34.089829 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:34.089829 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:34.089829 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:34.089829 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:34.089829 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:34.089829 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:34.089829 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:34.089829 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:34.089829 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:34.089829 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:34.089829 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:34.089829 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:34.091091 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:34.091091 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:34.091091 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:34.091091 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:34.091091 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 20 08:48:34.091091 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 20 08:48:34.091091 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 20 08:48:34.091091 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 20 08:48:34.091091 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 20 08:48:34.091091 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 20 08:48:34.091091 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 20 08:48:34.091091 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:34.091091 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 225/449] 10.222.0.36 () {56 vars in 1102 bytes} [Thu Oct 20 08:48:33 2022] GET /identity/v3/roles/service => generated 85 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:34.091091 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5af3476c-9969-48e0-b270-e5d6520001e1 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:34.094410 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebd695b5-3e98-4878-95bb-833125a8e0ab None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:34.095972 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebd695b5-3e98-4878-95bb-833125a8e0ab None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:34.109920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-ebd695b5-3e98-4878-95bb-833125a8e0ab None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:34.109920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:34.109920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:34.109920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:34.109920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:34.109920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:34.109920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:34.109920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:34.109920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:34.109920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:34.109920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:34.109920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:34.109920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 20 08:48:34.109920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 20 08:48:34.109920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 20 08:48:34.109920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 20 08:48:34.110941 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:34.110941 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:34.110941 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:34.110941 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:34.110941 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:34.110941 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:34.110941 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:34.110941 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:34.110941 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:34.110941 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:34.110941 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:34.110941 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:34.110941 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:34.110941 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:34.110941 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:34.110941 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:34.110941 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 20 08:48:34.111785 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 20 08:48:34.111785 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:34.111785 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:48:34.111785 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 225/450] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:48:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:34.111785 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebd695b5-3e98-4878-95bb-833125a8e0ab None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:34.190440 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:34.191706 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5af3476c-9969-48e0-b270-e5d6520001e1 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:34.210081 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebd695b5-3e98-4878-95bb-833125a8e0ab None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:34.225989 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55992025-cd72-420a-89f1-63b43e964cc3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:34.226673 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55992025-cd72-420a-89f1-63b43e964cc3 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:34.226888 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55992025-cd72-420a-89f1-63b43e964cc3 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:34.227152 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55992025-cd72-420a-89f1-63b43e964cc3 None admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:34.227775 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55992025-cd72-420a-89f1-63b43e964cc3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:34.228067 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55992025-cd72-420a-89f1-63b43e964cc3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:34.229431 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55992025-cd72-420a-89f1-63b43e964cc3 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:34.240785 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 226/451] 10.222.0.36 () {56 vars in 1109 bytes} [Thu Oct 20 08:48:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:34.241555 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55992025-cd72-420a-89f1-63b43e964cc3 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:34.247566 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b48e8d3b-8f8a-4389-86c7-7adf943ea75c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:34.248202 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b48e8d3b-8f8a-4389-86c7-7adf943ea75c None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:34.248482 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b48e8d3b-8f8a-4389-86c7-7adf943ea75c None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:34.248760 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b48e8d3b-8f8a-4389-86c7-7adf943ea75c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:34.296490 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:34.314201 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b48e8d3b-8f8a-4389-86c7-7adf943ea75c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:34.315701 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b48e8d3b-8f8a-4389-86c7-7adf943ea75c None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:34.346701 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55992025-cd72-420a-89f1-63b43e964cc3 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:34.382873 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194ba270-847a-440e-8c40-c3314beea472 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:34.383466 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194ba270-847a-440e-8c40-c3314beea472 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:34.383738 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194ba270-847a-440e-8c40-c3314beea472 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:34.384002 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194ba270-847a-440e-8c40-c3314beea472 None admin] PATH_INFO: `/v3/projects` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:34.384680 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-194ba270-847a-440e-8c40-c3314beea472 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:34.384977 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-194ba270-847a-440e-8c40-c3314beea472 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:34.386407 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-194ba270-847a-440e-8c40-c3314beea472 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:34.395365 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:34.405124 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 227/452] 10.222.0.36 () {56 vars in 1148 bytes} [Thu Oct 20 08:48:34 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:34.405930 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194ba270-847a-440e-8c40-c3314beea472 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:34.419236 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 226/453] 10.222.0.36 () {58 vars in 1282 bytes} [Thu Oct 20 08:48:34 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 303 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 20 08:48:34.419236 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b48e8d3b-8f8a-4389-86c7-7adf943ea75c None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:34.508766 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194ba270-847a-440e-8c40-c3314beea472 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:34.510661 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b48e8d3b-8f8a-4389-86c7-7adf943ea75c None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:34.518626 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:34.542872 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-352f5d5f-f00e-4659-8176-e0aabc28cc89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:34.543463 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-352f5d5f-f00e-4659-8176-e0aabc28cc89 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:34.543736 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-352f5d5f-f00e-4659-8176-e0aabc28cc89 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:34.544000 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-352f5d5f-f00e-4659-8176-e0aabc28cc89 None admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:34.544198 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f49e7b5d-2689-4af2-a1bd-9aecc4d4488d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:34.544646 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-352f5d5f-f00e-4659-8176-e0aabc28cc89 None admin] RBAC: Adding query filter params (name=service) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:34.544796 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49e7b5d-2689-4af2-a1bd-9aecc4d4488d None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:34.544941 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-352f5d5f-f00e-4659-8176-e0aabc28cc89 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:34.545074 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49e7b5d-2689-4af2-a1bd-9aecc4d4488d None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:34.545344 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49e7b5d-2689-4af2-a1bd-9aecc4d4488d None admin] PATH_INFO: `/v3/domains` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:34.545975 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f49e7b5d-2689-4af2-a1bd-9aecc4d4488d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:34.546274 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f49e7b5d-2689-4af2-a1bd-9aecc4d4488d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:34.546362 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-352f5d5f-f00e-4659-8176-e0aabc28cc89 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:34.547675 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f49e7b5d-2689-4af2-a1bd-9aecc4d4488d None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:34.554596 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 228/454] 10.222.0.36 () {56 vars in 1103 bytes} [Thu Oct 20 08:48:34 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:34.555365 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-352f5d5f-f00e-4659-8176-e0aabc28cc89 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:34.563358 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 227/455] 10.222.0.36 () {56 vars in 1109 bytes} [Thu Oct 20 08:48:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:34.564248 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f49e7b5d-2689-4af2-a1bd-9aecc4d4488d None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:34.608515 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:34.653890 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-352f5d5f-f00e-4659-8176-e0aabc28cc89 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:34.666841 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f49e7b5d-2689-4af2-a1bd-9aecc4d4488d None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:34.688016 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1936f091-e8ec-4291-b4bc-d85807b4b804 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:34.688612 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1936f091-e8ec-4291-b4bc-d85807b4b804 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:34.688888 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1936f091-e8ec-4291-b4bc-d85807b4b804 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:34.689156 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1936f091-e8ec-4291-b4bc-d85807b4b804 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:34.697982 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1936f091-e8ec-4291-b4bc-d85807b4b804 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:34.699016 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9c4cf6e-6eec-449f-bb62-1aaef35143e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:34.699451 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1936f091-e8ec-4291-b4bc-d85807b4b804 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:34.699656 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c4cf6e-6eec-449f-bb62-1aaef35143e9 None admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:34.699931 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c4cf6e-6eec-449f-bb62-1aaef35143e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:34.700202 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c4cf6e-6eec-449f-bb62-1aaef35143e9 None admin] PATH_INFO: `/v3/projects/d9857b610fb542969cfb5efbbdb8ce61/users/4d558918cd8b4ebea4fe8765297304db/roles/f81100d13381464aa48cbae3561ca1bc` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:34.711280 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-1936f091-e8ec-4291-b4bc-d85807b4b804 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Oct 20 08:48:34.711280 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:34.711280 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:34.711280 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:34.711280 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:34.711280 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:34.711280 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:34.711280 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:34.711280 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:34.711280 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:34.711280 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:34.711280 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:34.711280 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 20 08:48:34.711280 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 20 08:48:34.711280 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 20 08:48:34.711280 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 20 08:48:34.712239 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:34.712239 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:34.712239 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 20 08:48:34.712239 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 20 08:48:34.712239 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 20 08:48:34.712239 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 20 08:48:34.712239 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:34.712239 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:34.712239 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:34.712239 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:34.712239 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:34.712239 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:34.712239 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:34.712239 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:34.712239 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:34.712239 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:34.712239 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:34.712239 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:34.713144 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:34.713144 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:34.713144 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 20 08:48:34.713144 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 20 08:48:34.713144 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 20 08:48:34.713144 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 20 08:48:34.713144 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 20 08:48:34.713144 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 20 08:48:34.713144 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Oct 20 08:48:34.713144 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:34.713144 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 229/456] 10.222.0.36 () {56 vars in 1099 bytes} [Thu Oct 20 08:48:34 2022] GET /identity/v3/users/glance => generated 84 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:34.713713 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1936f091-e8ec-4291-b4bc-d85807b4b804 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:34.719918 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9c4cf6e-6eec-449f-bb62-1aaef35143e9 None admin] RBAC: Authorizing `identity:create_grant(project_id=d9857b610fb542969cfb5efbbdb8ce61, user_id=4d558918cd8b4ebea4fe8765297304db, role_id=f81100d13381464aa48cbae3561ca1bc)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:34.721381 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9c4cf6e-6eec-449f-bb62-1aaef35143e9 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:34.735629 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 228/457] 10.222.0.36 () {58 vars in 1439 bytes} [Thu Oct 20 08:48:34 2022] PUT /identity/v3/projects/d9857b610fb542969cfb5efbbdb8ce61/users/4d558918cd8b4ebea4fe8765297304db/roles/f81100d13381464aa48cbae3561ca1bc => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 08:48:34.736412 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9c4cf6e-6eec-449f-bb62-1aaef35143e9 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:34.763447 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:34.847265 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1936f091-e8ec-4291-b4bc-d85807b4b804 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:34.849810 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:34.860942 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9c4cf6e-6eec-449f-bb62-1aaef35143e9 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:34.887373 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75880112-9273-4653-b4bc-986dc35264bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:34.888019 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75880112-9273-4653-b4bc-986dc35264bb None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:34.888299 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75880112-9273-4653-b4bc-986dc35264bb None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:34.888567 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75880112-9273-4653-b4bc-986dc35264bb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:34.900520 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52521298-a389-4e64-9416-09eaa50a332f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:34.900732 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75880112-9273-4653-b4bc-986dc35264bb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:34.901148 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52521298-a389-4e64-9416-09eaa50a332f None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:34.901419 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52521298-a389-4e64-9416-09eaa50a332f None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:34.901688 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52521298-a389-4e64-9416-09eaa50a332f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:34.901996 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75880112-9273-4653-b4bc-986dc35264bb None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:34.914502 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52521298-a389-4e64-9416-09eaa50a332f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:34.915979 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52521298-a389-4e64-9416-09eaa50a332f None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:34.918035 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-75880112-9273-4653-b4bc-986dc35264bb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:34.918035 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:34.918035 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:34.918035 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:34.918035 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:34.918035 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:34.918035 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:34.918035 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:34.918035 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:34.918035 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:34.918035 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:34.918035 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:34.918035 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 20 08:48:34.918035 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 20 08:48:34.918035 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 20 08:48:34.918035 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 20 08:48:34.918931 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:34.918931 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:34.918931 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:34.918931 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:34.918931 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:34.918931 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:34.918931 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:34.918931 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:34.918931 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:34.918931 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:34.918931 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:34.918931 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:34.918931 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:34.918931 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:34.918931 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:34.918931 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:34.918931 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 20 08:48:34.919961 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 20 08:48:34.919961 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:34.919961 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:34.919961 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 230/458] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:48:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:34.920291 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75880112-9273-4653-b4bc-986dc35264bb None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:34.937261 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-52521298-a389-4e64-9416-09eaa50a332f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:34.937261 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:34.937261 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:34.937261 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:34.937261 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:34.937261 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:34.937261 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:34.937261 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:34.937261 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:34.937261 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:34.937261 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:34.937261 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:34.937261 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 20 08:48:34.937261 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 20 08:48:34.937261 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 20 08:48:34.937261 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 20 08:48:34.938239 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:34.938239 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:34.938239 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:34.938239 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:34.938239 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:34.938239 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:34.938239 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:34.938239 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:34.938239 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:34.938239 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:34.938239 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:34.938239 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:34.938239 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:34.938239 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:34.938239 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:34.938239 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:34.938239 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 20 08:48:34.939111 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 20 08:48:34.939111 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:34.939111 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:48:34.939111 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 229/459] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:48:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:34.939323 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52521298-a389-4e64-9416-09eaa50a332f None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:35.017769 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:35.036122 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75880112-9273-4653-b4bc-986dc35264bb None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:35.043124 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52521298-a389-4e64-9416-09eaa50a332f None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:35.071949 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-811b463d-db84-464a-921e-1ed2665e01d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:35.072627 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-811b463d-db84-464a-921e-1ed2665e01d7 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:35.072947 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-811b463d-db84-464a-921e-1ed2665e01d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:35.073189 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-811b463d-db84-464a-921e-1ed2665e01d7 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:35.078251 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d17fe228-c586-454f-86ba-ddc322f09d25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:35.078868 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17fe228-c586-454f-86ba-ddc322f09d25 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:35.079143 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17fe228-c586-454f-86ba-ddc322f09d25 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:35.079408 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17fe228-c586-454f-86ba-ddc322f09d25 None admin] PATH_INFO: `/v3/users` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:35.080137 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d17fe228-c586-454f-86ba-ddc322f09d25 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:35.080417 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d17fe228-c586-454f-86ba-ddc322f09d25 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:35.081023 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-811b463d-db84-464a-921e-1ed2665e01d7 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:35.081912 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d17fe228-c586-454f-86ba-ddc322f09d25 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:35.082440 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-811b463d-db84-464a-921e-1ed2665e01d7 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:35.095384 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-811b463d-db84-464a-921e-1ed2665e01d7 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 20 08:48:35.095384 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:35.095384 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:35.095384 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:35.095384 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:35.095384 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:35.095384 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:35.095384 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:35.095384 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:35.095384 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:35.095384 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:35.095384 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:35.095384 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 20 08:48:35.095384 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 20 08:48:35.095384 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 20 08:48:35.095384 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 20 08:48:35.096301 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:35.096301 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:35.096301 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 20 08:48:35.096301 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 20 08:48:35.096301 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 20 08:48:35.096301 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 20 08:48:35.096301 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:35.096301 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:35.096301 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:35.096301 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:35.096301 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:35.096301 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:35.096301 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:35.096301 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:35.096301 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:35.096301 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:35.096301 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:35.096301 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:35.102815 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:35.102815 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:35.102815 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 20 08:48:35.102815 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 20 08:48:35.102815 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 20 08:48:35.102815 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 20 08:48:35.102815 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 20 08:48:35.102815 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 20 08:48:35.102815 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 20 08:48:35.102815 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:35.102815 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 231/460] 10.222.0.36 () {56 vars in 1099 bytes} [Thu Oct 20 08:48:34 2022] GET /identity/v3/users/cinder => generated 84 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:35.102815 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-811b463d-db84-464a-921e-1ed2665e01d7 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:35.104772 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:35.112361 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 230/461] 10.222.0.36 () {56 vars in 1137 bytes} [Thu Oct 20 08:48:34 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:35.112361 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d17fe228-c586-454f-86ba-ddc322f09d25 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:35.214632 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-811b463d-db84-464a-921e-1ed2665e01d7 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:35.215187 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d17fe228-c586-454f-86ba-ddc322f09d25 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:35.249783 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db7044cd-2f2a-419d-83a4-0d81729f844f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:35.250382 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db7044cd-2f2a-419d-83a4-0d81729f844f None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:35.250685 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db7044cd-2f2a-419d-83a4-0d81729f844f None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:35.250955 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db7044cd-2f2a-419d-83a4-0d81729f844f None admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:35.251266 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e281b54f-fdeb-4145-9fe4-31c5d87cbf3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:35.251592 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db7044cd-2f2a-419d-83a4-0d81729f844f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:35.251859 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e281b54f-fdeb-4145-9fe4-31c5d87cbf3e None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:35.251961 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db7044cd-2f2a-419d-83a4-0d81729f844f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:35.252132 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e281b54f-fdeb-4145-9fe4-31c5d87cbf3e None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:35.252398 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e281b54f-fdeb-4145-9fe4-31c5d87cbf3e None admin] PATH_INFO: `/v3/domains` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:35.253025 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e281b54f-fdeb-4145-9fe4-31c5d87cbf3e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:35.253238 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db7044cd-2f2a-419d-83a4-0d81729f844f None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:35.253329 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e281b54f-fdeb-4145-9fe4-31c5d87cbf3e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:35.254777 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e281b54f-fdeb-4145-9fe4-31c5d87cbf3e None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:35.269525 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 232/462] 10.222.0.36 () {56 vars in 1109 bytes} [Thu Oct 20 08:48:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:35.269525 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db7044cd-2f2a-419d-83a4-0d81729f844f None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:35.269755 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 231/463] 10.222.0.36 () {56 vars in 1109 bytes} [Thu Oct 20 08:48:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:35.269755 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e281b54f-fdeb-4145-9fe4-31c5d87cbf3e None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:35.273971 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:35.330564 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:35.381281 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e281b54f-fdeb-4145-9fe4-31c5d87cbf3e None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:35.392665 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db7044cd-2f2a-419d-83a4-0d81729f844f None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:35.416573 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afcb5353-e4ff-4664-a5bd-458af971a16e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:35.417175 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afcb5353-e4ff-4664-a5bd-458af971a16e None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:35.417455 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afcb5353-e4ff-4664-a5bd-458af971a16e None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:35.417728 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afcb5353-e4ff-4664-a5bd-458af971a16e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:35.429301 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f25c961-6dfd-471b-aaf1-b7d651c7b1f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:35.429898 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f25c961-6dfd-471b-aaf1-b7d651c7b1f9 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:35.430176 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f25c961-6dfd-471b-aaf1-b7d651c7b1f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:35.430444 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f25c961-6dfd-471b-aaf1-b7d651c7b1f9 None admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:35.431198 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f25c961-6dfd-471b-aaf1-b7d651c7b1f9 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:35.431497 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f25c961-6dfd-471b-aaf1-b7d651c7b1f9 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:35.432929 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f25c961-6dfd-471b-aaf1-b7d651c7b1f9 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:35.456506 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 233/464] 10.222.0.36 () {56 vars in 1137 bytes} [Thu Oct 20 08:48:35 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:35.456506 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f25c961-6dfd-471b-aaf1-b7d651c7b1f9 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:35.497179 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afcb5353-e4ff-4664-a5bd-458af971a16e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:35.499528 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afcb5353-e4ff-4664-a5bd-458af971a16e None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:35.525978 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:35.558958 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:35.567269 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f25c961-6dfd-471b-aaf1-b7d651c7b1f9 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:35.602830 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b848d856-11a5-438e-bbfb-fb4467dd64ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:35.603449 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b848d856-11a5-438e-bbfb-fb4467dd64ee None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:35.603718 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b848d856-11a5-438e-bbfb-fb4467dd64ee None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:35.603979 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b848d856-11a5-438e-bbfb-fb4467dd64ee None admin] PATH_INFO: `/v3/projects/service` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:35.609803 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 232/465] 10.222.0.36 () {58 vars in 1282 bytes} [Thu Oct 20 08:48:35 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 338 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 20 08:48:35.609803 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afcb5353-e4ff-4664-a5bd-458af971a16e None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:35.616682 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b848d856-11a5-438e-bbfb-fb4467dd64ee None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:35.618995 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b848d856-11a5-438e-bbfb-fb4467dd64ee None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:35.633610 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-b848d856-11a5-438e-bbfb-fb4467dd64ee None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 20 08:48:35.633610 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:35.633610 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:35.633610 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:35.633610 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:35.633610 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:35.633610 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:35.633610 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:35.633610 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:35.633610 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:35.633610 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:35.633610 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:35.633610 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 20 08:48:35.633610 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 20 08:48:35.633610 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 20 08:48:35.634602 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 20 08:48:35.634602 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:35.634602 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:35.634602 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:35.634602 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:35.634602 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:35.634602 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:35.634602 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:35.634602 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:35.634602 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:35.634602 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:35.634602 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:35.634602 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:35.634602 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:35.634602 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:35.634602 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:35.634602 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:35.635475 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 20 08:48:35.635475 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 20 08:48:35.635475 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 20 08:48:35.635475 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 20 08:48:35.635475 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 20 08:48:35.635475 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 20 08:48:35.635475 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 20 08:48:35.635475 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:35.635475 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 234/466] 10.222.0.36 () {56 vars in 1111 bytes} [Thu Oct 20 08:48:35 2022] GET /identity/v3/projects/service => generated 88 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:35.635939 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b848d856-11a5-438e-bbfb-fb4467dd64ee None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:35.715615 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afcb5353-e4ff-4664-a5bd-458af971a16e None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:35.747518 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b848d856-11a5-438e-bbfb-fb4467dd64ee None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:35.748873 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0f2a0c7-333e-4d5b-bd9b-913d2b525bff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:35.750568 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f2a0c7-333e-4d5b-bd9b-913d2b525bff None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:35.750859 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f2a0c7-333e-4d5b-bd9b-913d2b525bff None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:35.751154 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f2a0c7-333e-4d5b-bd9b-913d2b525bff None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:35.759784 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0f2a0c7-333e-4d5b-bd9b-913d2b525bff None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:35.761479 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0f2a0c7-333e-4d5b-bd9b-913d2b525bff None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:35.770261 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:35.773377 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-b0f2a0c7-333e-4d5b-bd9b-913d2b525bff None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Oct 20 08:48:35.773377 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:35.773377 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:35.773377 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:35.773377 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:35.773377 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:35.773377 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:35.773377 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:35.773377 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:35.773377 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:35.773377 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:35.773377 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:35.773377 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 20 08:48:35.773377 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 20 08:48:35.773377 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 20 08:48:35.773377 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 20 08:48:35.774289 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:35.774289 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:35.774289 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 20 08:48:35.774289 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 20 08:48:35.774289 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 20 08:48:35.774289 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 20 08:48:35.774289 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:35.774289 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:35.774289 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:35.774289 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:35.774289 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:35.774289 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:35.774289 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:35.774289 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:35.774289 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:35.774289 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:35.774289 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:35.774289 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:35.775930 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:35.775930 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:35.775930 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 20 08:48:35.775930 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 20 08:48:35.775930 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 20 08:48:35.775930 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 20 08:48:35.775930 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 20 08:48:35.775930 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 20 08:48:35.775930 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Oct 20 08:48:35.775930 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:48:35.775930 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 233/467] 10.222.0.36 () {56 vars in 1102 bytes} [Thu Oct 20 08:48:35 2022] GET /identity/v3/users/neutron => generated 85 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:35.775930 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0f2a0c7-333e-4d5b-bd9b-913d2b525bff None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:35.788370 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e0e7621-0493-40e2-adac-cdd423ec6d4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:35.788907 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e0e7621-0493-40e2-adac-cdd423ec6d4f None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:35.789181 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e0e7621-0493-40e2-adac-cdd423ec6d4f None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:35.789444 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e0e7621-0493-40e2-adac-cdd423ec6d4f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:35.801132 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:35.861941 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e0e7621-0493-40e2-adac-cdd423ec6d4f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:35.863407 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e0e7621-0493-40e2-adac-cdd423ec6d4f None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:35.886754 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0f2a0c7-333e-4d5b-bd9b-913d2b525bff None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:35.921162 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4233c2d0-81b8-48d7-9ff4-b03c46dae23d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:35.921782 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4233c2d0-81b8-48d7-9ff4-b03c46dae23d None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:35.922056 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4233c2d0-81b8-48d7-9ff4-b03c46dae23d None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:35.922331 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4233c2d0-81b8-48d7-9ff4-b03c46dae23d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:35.934157 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4233c2d0-81b8-48d7-9ff4-b03c46dae23d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:35.936577 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4233c2d0-81b8-48d7-9ff4-b03c46dae23d None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:35.958707 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-4233c2d0-81b8-48d7-9ff4-b03c46dae23d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:35.958707 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:35.958707 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:35.958707 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:35.958707 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:35.958707 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:35.958707 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:35.958707 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:35.958707 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:35.958707 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:35.958707 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:35.958707 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:35.958707 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 20 08:48:35.958707 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 20 08:48:35.958707 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 20 08:48:35.958707 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 20 08:48:35.959660 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:35.959660 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:35.959660 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:35.959660 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:35.959660 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:35.959660 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:35.959660 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:35.959660 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:35.959660 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:35.959660 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:35.959660 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:35.959660 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:35.959660 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:35.959660 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:35.959660 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:35.959660 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:35.959660 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 20 08:48:35.960523 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 20 08:48:35.960523 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:35.960523 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:48:35.966939 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 234/468] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:48:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:35.968061 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4233c2d0-81b8-48d7-9ff4-b03c46dae23d None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:35.983122 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 235/469] 10.222.0.36 () {58 vars in 1282 bytes} [Thu Oct 20 08:48:35 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 343 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 20 08:48:35.983122 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e0e7621-0493-40e2-adac-cdd423ec6d4f None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:36.005749 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:36.067352 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:36.101230 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e0e7621-0493-40e2-adac-cdd423ec6d4f None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:36.106278 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4233c2d0-81b8-48d7-9ff4-b03c46dae23d None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:36.141334 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9bec6f1-6b8c-4f49-94e6-a23de016cbc6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:36.141924 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9bec6f1-6b8c-4f49-94e6-a23de016cbc6 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:36.142190 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9bec6f1-6b8c-4f49-94e6-a23de016cbc6 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:36.142450 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9bec6f1-6b8c-4f49-94e6-a23de016cbc6 None admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:36.143166 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9bec6f1-6b8c-4f49-94e6-a23de016cbc6 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:36.143468 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9bec6f1-6b8c-4f49-94e6-a23de016cbc6 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:36.144446 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-381126ef-0166-4e54-99ce-a2fd970b990d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:36.144957 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9bec6f1-6b8c-4f49-94e6-a23de016cbc6 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:36.145067 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-381126ef-0166-4e54-99ce-a2fd970b990d None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:36.145309 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-381126ef-0166-4e54-99ce-a2fd970b990d None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:36.145573 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-381126ef-0166-4e54-99ce-a2fd970b990d None admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:36.147414 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-381126ef-0166-4e54-99ce-a2fd970b990d None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:36.147716 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-381126ef-0166-4e54-99ce-a2fd970b990d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:36.149227 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-381126ef-0166-4e54-99ce-a2fd970b990d None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:36.157905 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 235/470] 10.222.0.36 () {56 vars in 1148 bytes} [Thu Oct 20 08:48:35 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:36.164181 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9bec6f1-6b8c-4f49-94e6-a23de016cbc6 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:36.173955 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 236/471] 10.222.0.36 () {56 vars in 1139 bytes} [Thu Oct 20 08:48:35 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:36.174914 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-381126ef-0166-4e54-99ce-a2fd970b990d None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:36.274178 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:36.297558 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-381126ef-0166-4e54-99ce-a2fd970b990d None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:36.303151 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9bec6f1-6b8c-4f49-94e6-a23de016cbc6 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:36.343475 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1b2d231-5ee7-45a4-8047-908e4da3b5c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:36.344477 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1b2d231-5ee7-45a4-8047-908e4da3b5c4 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:36.345144 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1b2d231-5ee7-45a4-8047-908e4da3b5c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:36.345622 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1b2d231-5ee7-45a4-8047-908e4da3b5c4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:36.348503 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:36.356600 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf074c65-a3f1-4af6-9e9e-98e3be26c3f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:36.356971 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf074c65-a3f1-4af6-9e9e-98e3be26c3f5 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:36.357270 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf074c65-a3f1-4af6-9e9e-98e3be26c3f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:36.357541 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf074c65-a3f1-4af6-9e9e-98e3be26c3f5 None admin] PATH_INFO: `/v3/domains` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:36.358183 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf074c65-a3f1-4af6-9e9e-98e3be26c3f5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:36.358509 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf074c65-a3f1-4af6-9e9e-98e3be26c3f5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:36.359755 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1b2d231-5ee7-45a4-8047-908e4da3b5c4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:36.359881 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf074c65-a3f1-4af6-9e9e-98e3be26c3f5 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:36.361631 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1b2d231-5ee7-45a4-8047-908e4da3b5c4 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:36.380070 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-b1b2d231-5ee7-45a4-8047-908e4da3b5c4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:36.380070 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:36.380070 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:36.380070 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:36.380070 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:36.380070 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:36.380070 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:36.380070 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:36.380070 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:36.380070 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:36.380070 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:36.380070 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:36.380070 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 20 08:48:36.380070 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 20 08:48:36.380070 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 20 08:48:36.380070 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 20 08:48:36.381565 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:36.381565 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:36.381565 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:36.381565 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:36.381565 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:36.381565 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:36.381565 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:36.381565 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:36.381565 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:36.381565 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:36.381565 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:36.381565 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:36.381565 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:36.381565 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:36.381565 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:36.381565 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:36.381565 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 20 08:48:36.382430 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 20 08:48:36.382430 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:36.382430 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:36.382430 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 237/472] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:48:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:36.383335 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1b2d231-5ee7-45a4-8047-908e4da3b5c4 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:36.387188 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 236/473] 10.222.0.36 () {56 vars in 1109 bytes} [Thu Oct 20 08:48:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:36.387838 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf074c65-a3f1-4af6-9e9e-98e3be26c3f5 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:36.492881 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1b2d231-5ee7-45a4-8047-908e4da3b5c4 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:36.493401 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf074c65-a3f1-4af6-9e9e-98e3be26c3f5 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:36.523453 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:36.529071 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-026a8597-983b-4560-9fd0-0bbdf2a76d39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:36.529220 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18ae0078-f349-41d1-8e0a-ca2941651356 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:36.529669 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-026a8597-983b-4560-9fd0-0bbdf2a76d39 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:36.529772 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18ae0078-f349-41d1-8e0a-ca2941651356 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:36.529940 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-026a8597-983b-4560-9fd0-0bbdf2a76d39 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:36.530036 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18ae0078-f349-41d1-8e0a-ca2941651356 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:36.530206 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18ae0078-f349-41d1-8e0a-ca2941651356 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:36.530310 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-026a8597-983b-4560-9fd0-0bbdf2a76d39 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:36.531400 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18ae0078-f349-41d1-8e0a-ca2941651356 None admin] RBAC: Adding query filter params (role.id=f81100d13381464aa48cbae3561ca1bc, scope.project.id=d9857b610fb542969cfb5efbbdb8ce61, user.id=a94e868115ec49918eda10724263930c) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:36.531704 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18ae0078-f349-41d1-8e0a-ca2941651356 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:36.533121 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18ae0078-f349-41d1-8e0a-ca2941651356 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:36.539915 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 238/474] 10.222.0.36 () {56 vars in 1374 bytes} [Thu Oct 20 08:48:36 2022] GET /identity/v3/role_assignments?user.id=a94e868115ec49918eda10724263930c&scope.project.id=d9857b610fb542969cfb5efbbdb8ce61&role.id=f81100d13381464aa48cbae3561ca1bc => generated 608 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:36.540466 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18ae0078-f349-41d1-8e0a-ca2941651356 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:36.629706 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:36.646116 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-026a8597-983b-4560-9fd0-0bbdf2a76d39 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:36.648866 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-026a8597-983b-4560-9fd0-0bbdf2a76d39 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:36.656941 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18ae0078-f349-41d1-8e0a-ca2941651356 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:36.697003 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d3092b1-c09a-4ee3-b30c-35bc360bc14e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:36.697607 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d3092b1-c09a-4ee3-b30c-35bc360bc14e None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:36.697880 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d3092b1-c09a-4ee3-b30c-35bc360bc14e None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:36.698378 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d3092b1-c09a-4ee3-b30c-35bc360bc14e None admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:36.701124 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d3092b1-c09a-4ee3-b30c-35bc360bc14e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:36.701426 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d3092b1-c09a-4ee3-b30c-35bc360bc14e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:36.702862 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d3092b1-c09a-4ee3-b30c-35bc360bc14e None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:36.717904 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 239/475] 10.222.0.36 () {56 vars in 1109 bytes} [Thu Oct 20 08:48:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:36.718964 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d3092b1-c09a-4ee3-b30c-35bc360bc14e None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:36.790766 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 237/476] 10.222.0.36 () {58 vars in 1282 bytes} [Thu Oct 20 08:48:36 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 403 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 20 08:48:36.791496 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-026a8597-983b-4560-9fd0-0bbdf2a76d39 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:36.796997 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:36.850280 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d3092b1-c09a-4ee3-b30c-35bc360bc14e None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:36.888349 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3861facb-4480-4d72-8df7-6a5d83208089 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:36.889247 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3861facb-4480-4d72-8df7-6a5d83208089 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:36.889705 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3861facb-4480-4d72-8df7-6a5d83208089 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:36.890140 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3861facb-4480-4d72-8df7-6a5d83208089 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:36.898345 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3861facb-4480-4d72-8df7-6a5d83208089 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:36.899847 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3861facb-4480-4d72-8df7-6a5d83208089 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:36.911233 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-3861facb-4480-4d72-8df7-6a5d83208089 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 20 08:48:36.911233 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:36.911233 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:36.911233 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:36.911233 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:36.911233 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:36.911233 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:36.911233 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:36.911233 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:36.911233 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:36.911233 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:36.911233 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:36.911233 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 20 08:48:36.911233 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 20 08:48:36.911233 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 20 08:48:36.912666 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 20 08:48:36.912666 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:36.912666 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:36.912666 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:36.912666 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:36.912666 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:36.912666 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:36.912666 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:36.912666 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:36.912666 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:36.912666 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:36.912666 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:36.912666 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:36.912666 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:36.912666 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:36.912666 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:36.912666 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:36.913540 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 20 08:48:36.913540 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 20 08:48:36.913540 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 20 08:48:36.913540 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 20 08:48:36.913540 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 20 08:48:36.913540 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 20 08:48:36.913540 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 20 08:48:36.913540 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:36.913540 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 240/477] 10.222.0.36 () {56 vars in 1111 bytes} [Thu Oct 20 08:48:36 2022] GET /identity/v3/projects/service => generated 88 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:36.914000 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3861facb-4480-4d72-8df7-6a5d83208089 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:36.922637 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:36.931886 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-026a8597-983b-4560-9fd0-0bbdf2a76d39 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:36.967479 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2879b687-9d07-4097-adff-513ddb60a123 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:36.968100 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2879b687-9d07-4097-adff-513ddb60a123 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:36.968374 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2879b687-9d07-4097-adff-513ddb60a123 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:36.968639 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2879b687-9d07-4097-adff-513ddb60a123 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:36.975965 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2879b687-9d07-4097-adff-513ddb60a123 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:36.977506 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2879b687-9d07-4097-adff-513ddb60a123 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:36.989117 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-2879b687-9d07-4097-adff-513ddb60a123 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 20 08:48:36.989117 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:36.989117 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:36.989117 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:36.989117 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:36.989117 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:36.989117 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:36.989117 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:36.989117 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:36.989117 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:36.989117 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:36.989117 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:36.989117 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 20 08:48:36.989117 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 20 08:48:36.989117 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 20 08:48:36.990022 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 20 08:48:36.990022 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:36.990022 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:36.990022 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:36.990022 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:36.990022 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:36.990022 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:36.990022 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:36.990022 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:36.990022 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:36.990022 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:36.990022 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:36.990022 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:36.990022 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:36.990022 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:36.990022 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:36.990022 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:36.991297 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 20 08:48:36.991297 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 20 08:48:36.991297 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 20 08:48:36.991297 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 20 08:48:36.991297 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 20 08:48:36.991297 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 20 08:48:36.991297 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 20 08:48:36.991297 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:48:36.994851 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 238/478] 10.222.0.36 () {56 vars in 1111 bytes} [Thu Oct 20 08:48:36 2022] GET /identity/v3/projects/service => generated 88 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:36.996223 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb1e0a7-dc30-49aa-96cc-7365f5bc598b None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:36.996504 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb1e0a7-dc30-49aa-96cc-7365f5bc598b None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:36.996776 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb1e0a7-dc30-49aa-96cc-7365f5bc598b None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:36.998306 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 239/479] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:48:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:48:36.999028 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbb1e0a7-dc30-49aa-96cc-7365f5bc598b None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:37.022924 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3861facb-4480-4d72-8df7-6a5d83208089 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:37.039746 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:37.059754 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a572bc5a-3a4c-42b4-82ff-21e4291f0446 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:37.060375 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a572bc5a-3a4c-42b4-82ff-21e4291f0446 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:37.061183 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a572bc5a-3a4c-42b4-82ff-21e4291f0446 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:37.061453 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a572bc5a-3a4c-42b4-82ff-21e4291f0446 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:37.075391 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a572bc5a-3a4c-42b4-82ff-21e4291f0446 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:37.077338 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a572bc5a-3a4c-42b4-82ff-21e4291f0446 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:37.094373 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-a572bc5a-3a4c-42b4-82ff-21e4291f0446 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:37.094373 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:37.094373 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:37.094373 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:37.094373 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:37.094373 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:37.094373 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:37.094373 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:37.094373 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:37.094373 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:37.094373 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:37.094373 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:37.094373 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 20 08:48:37.094373 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 20 08:48:37.094373 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 20 08:48:37.094373 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 20 08:48:37.095461 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:37.095461 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:37.095461 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:37.095461 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:37.095461 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:37.095461 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:37.095461 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:37.095461 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:37.095461 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:37.095461 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:37.095461 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:37.095461 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:37.095461 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:37.095461 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:37.095461 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:37.095461 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:37.095461 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 20 08:48:37.101722 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 20 08:48:37.101722 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:37.101722 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:37.101722 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 241/480] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:48:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:37.101722 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a572bc5a-3a4c-42b4-82ff-21e4291f0446 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:37.117191 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbb1e0a7-dc30-49aa-96cc-7365f5bc598b None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:37.154598 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72e29458-90b2-4348-8595-4c1475392c83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:37.155133 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72e29458-90b2-4348-8595-4c1475392c83 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:37.155417 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72e29458-90b2-4348-8595-4c1475392c83 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:37.155689 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72e29458-90b2-4348-8595-4c1475392c83 None admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:37.156372 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72e29458-90b2-4348-8595-4c1475392c83 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:37.156695 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72e29458-90b2-4348-8595-4c1475392c83 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:37.158123 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72e29458-90b2-4348-8595-4c1475392c83 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:37.175168 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 240/481] 10.222.0.36 () {56 vars in 1148 bytes} [Thu Oct 20 08:48:36 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:37.175168 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6adf56c8-22c5-4424-acfd-b9cd1458d70a None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:37.175168 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6adf56c8-22c5-4424-acfd-b9cd1458d70a None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:37.175168 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6adf56c8-22c5-4424-acfd-b9cd1458d70a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:37.193746 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-6adf56c8-22c5-4424-acfd-b9cd1458d70a None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:48:37.195283 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-6adf56c8-22c5-4424-acfd-b9cd1458d70a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:48:37.204790 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:37.213023 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a572bc5a-3a4c-42b4-82ff-21e4291f0446 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:37.248098 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11fe9baa-8446-4fbd-bc61-22774daf0dfe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:37.248962 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11fe9baa-8446-4fbd-bc61-22774daf0dfe None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:37.249240 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11fe9baa-8446-4fbd-bc61-22774daf0dfe None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:37.249592 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11fe9baa-8446-4fbd-bc61-22774daf0dfe None admin] PATH_INFO: `/v3/projects` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:37.250193 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11fe9baa-8446-4fbd-bc61-22774daf0dfe None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:37.250527 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11fe9baa-8446-4fbd-bc61-22774daf0dfe None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:37.251944 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11fe9baa-8446-4fbd-bc61-22774daf0dfe None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:37.270331 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 242/482] 10.222.0.36 () {56 vars in 1148 bytes} [Thu Oct 20 08:48:37 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:37.270331 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11fe9baa-8446-4fbd-bc61-22774daf0dfe None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:37.273118 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 241/483] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:48:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 20 08:48:37.275375 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6adf56c8-22c5-4424-acfd-b9cd1458d70a None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:37.308682 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:37.401986 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11fe9baa-8446-4fbd-bc61-22774daf0dfe None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:37.406436 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6adf56c8-22c5-4424-acfd-b9cd1458d70a None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:37.441465 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3170f4e0-e3a1-4bdb-8449-42650460d387 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:37.442375 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3170f4e0-e3a1-4bdb-8449-42650460d387 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:37.443395 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3170f4e0-e3a1-4bdb-8449-42650460d387 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:37.443851 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3170f4e0-e3a1-4bdb-8449-42650460d387 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:37.444705 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f209685f-6d59-4191-9c23-eaea22ba2aed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:37.444966 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3170f4e0-e3a1-4bdb-8449-42650460d387 None admin] RBAC: Adding query filter params (role.id=f81100d13381464aa48cbae3561ca1bc, scope.project.id=d9857b610fb542969cfb5efbbdb8ce61, user.id=ce043914cd9042909f70ab0adb272d34) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:37.445436 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f209685f-6d59-4191-9c23-eaea22ba2aed None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:37.445604 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3170f4e0-e3a1-4bdb-8449-42650460d387 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:37.445824 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f209685f-6d59-4191-9c23-eaea22ba2aed None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:37.446160 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f209685f-6d59-4191-9c23-eaea22ba2aed None admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:37.446937 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f209685f-6d59-4191-9c23-eaea22ba2aed None admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:37.447308 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f209685f-6d59-4191-9c23-eaea22ba2aed None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:37.448102 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3170f4e0-e3a1-4bdb-8449-42650460d387 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:37.448787 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f209685f-6d59-4191-9c23-eaea22ba2aed None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:37.459599 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 242/484] 10.222.0.36 () {56 vars in 1374 bytes} [Thu Oct 20 08:48:37 2022] GET /identity/v3/role_assignments?user.id=ce043914cd9042909f70ab0adb272d34&scope.project.id=d9857b610fb542969cfb5efbbdb8ce61&role.id=f81100d13381464aa48cbae3561ca1bc => generated 608 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:37.460763 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3170f4e0-e3a1-4bdb-8449-42650460d387 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:37.466532 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 243/485] 10.222.0.36 () {56 vars in 1109 bytes} [Thu Oct 20 08:48:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:37.468264 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f209685f-6d59-4191-9c23-eaea22ba2aed None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:37.477470 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:37.600147 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:37.613552 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f209685f-6d59-4191-9c23-eaea22ba2aed None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:37.650402 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3170f4e0-e3a1-4bdb-8449-42650460d387 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:37.652978 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4cbe2a7-8565-434b-b4fd-759db4d5d9fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:37.653617 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4cbe2a7-8565-434b-b4fd-759db4d5d9fa None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:37.653897 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4cbe2a7-8565-434b-b4fd-759db4d5d9fa None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:37.654168 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4cbe2a7-8565-434b-b4fd-759db4d5d9fa None admin] PATH_INFO: `/v3/projects/service` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:37.662608 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4cbe2a7-8565-434b-b4fd-759db4d5d9fa None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:37.664016 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4cbe2a7-8565-434b-b4fd-759db4d5d9fa None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:37.676941 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-c4cbe2a7-8565-434b-b4fd-759db4d5d9fa None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 20 08:48:37.676941 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:37.676941 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:37.676941 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:37.676941 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:37.676941 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:37.676941 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:37.676941 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:37.676941 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:37.676941 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:37.676941 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:37.676941 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:37.676941 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 20 08:48:37.676941 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 20 08:48:37.676941 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 20 08:48:37.677984 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 20 08:48:37.677984 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:37.677984 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:37.677984 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:37.677984 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:37.677984 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:37.677984 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:37.677984 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:37.677984 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:37.677984 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:37.677984 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:37.677984 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:37.677984 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:37.677984 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:37.677984 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:37.677984 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:37.677984 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:37.678886 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 20 08:48:37.678886 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 20 08:48:37.678886 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 20 08:48:37.678886 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 20 08:48:37.678886 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 20 08:48:37.678886 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 20 08:48:37.678886 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 20 08:48:37.678886 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:37.678886 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 244/486] 10.222.0.36 () {56 vars in 1111 bytes} [Thu Oct 20 08:48:37 2022] GET /identity/v3/projects/service => generated 88 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:37.679345 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4cbe2a7-8565-434b-b4fd-759db4d5d9fa None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:37.695349 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7791d65-c036-4619-8e5e-0322da7c9f83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:37.696455 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7791d65-c036-4619-8e5e-0322da7c9f83 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:37.696834 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7791d65-c036-4619-8e5e-0322da7c9f83 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:37.697208 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7791d65-c036-4619-8e5e-0322da7c9f83 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:37.698339 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7791d65-c036-4619-8e5e-0322da7c9f83 None admin] RBAC: Adding query filter params (role.id=f81100d13381464aa48cbae3561ca1bc, scope.project.id=d9857b610fb542969cfb5efbbdb8ce61, user.id=7cfa6bb090de44eaa5bdf099a3096e8c) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:37.698747 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7791d65-c036-4619-8e5e-0322da7c9f83 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:37.700259 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7791d65-c036-4619-8e5e-0322da7c9f83 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:37.707286 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 243/487] 10.222.0.36 () {56 vars in 1374 bytes} [Thu Oct 20 08:48:37 2022] GET /identity/v3/role_assignments?user.id=7cfa6bb090de44eaa5bdf099a3096e8c&scope.project.id=d9857b610fb542969cfb5efbbdb8ce61&role.id=f81100d13381464aa48cbae3561ca1bc => generated 608 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:37.708180 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7791d65-c036-4619-8e5e-0322da7c9f83 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:37.790283 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:37.821692 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4cbe2a7-8565-434b-b4fd-759db4d5d9fa None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:37.839063 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7791d65-c036-4619-8e5e-0322da7c9f83 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:37.868329 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c49f3de3-d2f4-4840-bfe6-5d10de602c0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:37.869532 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49f3de3-d2f4-4840-bfe6-5d10de602c0b None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:37.869532 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49f3de3-d2f4-4840-bfe6-5d10de602c0b None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:37.869953 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49f3de3-d2f4-4840-bfe6-5d10de602c0b None admin] PATH_INFO: `/v3/roles/service` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:37.884140 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c49f3de3-d2f4-4840-bfe6-5d10de602c0b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 20 08:48:37.887078 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-458df13b-9363-440e-bf7d-7a4d4677bfde None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:37.887866 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458df13b-9363-440e-bf7d-7a4d4677bfde None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:37.888286 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458df13b-9363-440e-bf7d-7a4d4677bfde None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:37.888699 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458df13b-9363-440e-bf7d-7a4d4677bfde None admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:37.889586 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-458df13b-9363-440e-bf7d-7a4d4677bfde None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:37.890042 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-458df13b-9363-440e-bf7d-7a4d4677bfde None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:37.891678 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-458df13b-9363-440e-bf7d-7a4d4677bfde None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:37.891994 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-c49f3de3-d2f4-4840-bfe6-5d10de602c0b None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 20 08:48:37.891994 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:37.891994 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:37.891994 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:37.891994 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:37.891994 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:37.891994 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:37.891994 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:37.891994 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:37.891994 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:37.891994 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:37.891994 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:37.891994 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 20 08:48:37.891994 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 20 08:48:37.891994 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 20 08:48:37.891994 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 20 08:48:37.893058 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 20 08:48:37.893058 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 20 08:48:37.893058 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 20 08:48:37.893058 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 20 08:48:37.893058 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:37.893058 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:37.893058 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:37.893058 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:37.893058 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:37.893058 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:37.893058 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:37.893058 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:37.893058 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:37.893058 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:37.893058 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:37.893058 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:37.894033 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:37.894033 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:37.894033 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:37.894033 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:37.894033 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 20 08:48:37.894033 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 20 08:48:37.894033 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 20 08:48:37.894033 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 20 08:48:37.894033 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 20 08:48:37.894033 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 20 08:48:37.894033 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 20 08:48:37.894033 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:37.895980 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 245/488] 10.222.0.36 () {56 vars in 1102 bytes} [Thu Oct 20 08:48:37 2022] GET /identity/v3/roles/service => generated 85 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:37.901860 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c49f3de3-d2f4-4840-bfe6-5d10de602c0b None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:37.910887 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 244/489] 10.222.0.36 () {56 vars in 1148 bytes} [Thu Oct 20 08:48:37 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:37.919715 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:37.920531 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-458df13b-9363-440e-bf7d-7a4d4677bfde None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:38.039377 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c49f3de3-d2f4-4840-bfe6-5d10de602c0b None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:38.042266 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-458df13b-9363-440e-bf7d-7a4d4677bfde None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:38.094818 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:38.101737 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a497d8bf-4dfb-466f-a62c-80c407f634e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:38.103252 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a497d8bf-4dfb-466f-a62c-80c407f634e5 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:38.104521 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33e052f5-db4e-4e52-abc9-f4553e975d22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:38.104915 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a497d8bf-4dfb-466f-a62c-80c407f634e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:38.105291 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a497d8bf-4dfb-466f-a62c-80c407f634e5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:38.105967 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33e052f5-db4e-4e52-abc9-f4553e975d22 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:38.106857 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a497d8bf-4dfb-466f-a62c-80c407f634e5 None admin] RBAC: Adding query filter params (role.id=f81100d13381464aa48cbae3561ca1bc, scope.project.id=d9857b610fb542969cfb5efbbdb8ce61, user.id=794ca389002d41748d7925216ed964a5) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:38.107234 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33e052f5-db4e-4e52-abc9-f4553e975d22 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:38.107638 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a497d8bf-4dfb-466f-a62c-80c407f634e5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:38.107998 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33e052f5-db4e-4e52-abc9-f4553e975d22 None admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:38.110231 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33e052f5-db4e-4e52-abc9-f4553e975d22 None admin] RBAC: Adding query filter params (name=service) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:38.110676 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33e052f5-db4e-4e52-abc9-f4553e975d22 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:38.112128 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33e052f5-db4e-4e52-abc9-f4553e975d22 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:38.112293 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a497d8bf-4dfb-466f-a62c-80c407f634e5 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:38.123619 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 245/490] 10.222.0.36 () {56 vars in 1374 bytes} [Thu Oct 20 08:48:37 2022] GET /identity/v3/role_assignments?user.id=794ca389002d41748d7925216ed964a5&scope.project.id=d9857b610fb542969cfb5efbbdb8ce61&role.id=f81100d13381464aa48cbae3561ca1bc => generated 608 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:38.127418 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 246/491] 10.222.0.36 () {56 vars in 1103 bytes} [Thu Oct 20 08:48:37 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 08:48:38.133324 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a497d8bf-4dfb-466f-a62c-80c407f634e5 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:38.287637 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a497d8bf-4dfb-466f-a62c-80c407f634e5 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:38.347818 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1296e2ea-b3ed-4393-80b4-006b515ca2ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:38.348455 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1296e2ea-b3ed-4393-80b4-006b515ca2ae None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:38.348733 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1296e2ea-b3ed-4393-80b4-006b515ca2ae None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:38.349076 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1296e2ea-b3ed-4393-80b4-006b515ca2ae None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:38.370963 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1296e2ea-b3ed-4393-80b4-006b515ca2ae None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:38.372442 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1296e2ea-b3ed-4393-80b4-006b515ca2ae None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:38.396369 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-1296e2ea-b3ed-4393-80b4-006b515ca2ae None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:38.396369 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:38.396369 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:38.396369 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:38.396369 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:38.396369 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:38.396369 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:38.396369 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:38.396369 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:38.396369 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:38.396369 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:38.396369 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:38.396369 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 20 08:48:38.396369 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 20 08:48:38.396369 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 20 08:48:38.396369 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 20 08:48:38.397506 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:38.397506 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:38.397506 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:38.397506 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:38.397506 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:38.397506 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:38.397506 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:38.397506 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:38.397506 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:38.397506 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:38.397506 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:38.397506 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:38.397506 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:38.397506 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:38.397506 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:38.397506 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:38.397506 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 20 08:48:38.399310 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 20 08:48:38.399310 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:38.399310 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:48:38.399310 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 246/492] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:48:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 265 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:38.403401 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33e052f5-db4e-4e52-abc9-f4553e975d22 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:38.512859 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:38.548317 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33e052f5-db4e-4e52-abc9-f4553e975d22 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:38.582700 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a13c61-1608-4efc-98cb-ef16174c673c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:38.584083 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a13c61-1608-4efc-98cb-ef16174c673c None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:38.584363 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a13c61-1608-4efc-98cb-ef16174c673c None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:38.584631 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a13c61-1608-4efc-98cb-ef16174c673c None admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:38.585273 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5a13c61-1608-4efc-98cb-ef16174c673c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:38.585567 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5a13c61-1608-4efc-98cb-ef16174c673c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:38.586967 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5a13c61-1608-4efc-98cb-ef16174c673c None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:38.599871 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 247/493] 10.222.0.36 () {56 vars in 1109 bytes} [Thu Oct 20 08:48:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:38.605518 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1296e2ea-b3ed-4393-80b4-006b515ca2ae None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:38.656704 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:38.716694 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1296e2ea-b3ed-4393-80b4-006b515ca2ae None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:38.750917 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dc68a49-3a19-4b42-9f56-2e707b155b52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:38.751547 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc68a49-3a19-4b42-9f56-2e707b155b52 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:38.751823 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc68a49-3a19-4b42-9f56-2e707b155b52 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:38.752092 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc68a49-3a19-4b42-9f56-2e707b155b52 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:38.761078 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dc68a49-3a19-4b42-9f56-2e707b155b52 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:38.762583 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dc68a49-3a19-4b42-9f56-2e707b155b52 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:38.777843 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-9dc68a49-3a19-4b42-9f56-2e707b155b52 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 20 08:48:38.777843 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:38.777843 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:38.777843 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:38.777843 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:38.777843 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:38.777843 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:38.777843 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:38.777843 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:38.777843 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:38.777843 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:38.777843 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:38.777843 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 20 08:48:38.777843 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 20 08:48:38.777843 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 20 08:48:38.777843 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 20 08:48:38.778873 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:38.778873 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:38.778873 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 20 08:48:38.778873 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 20 08:48:38.778873 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 20 08:48:38.778873 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 20 08:48:38.778873 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:38.778873 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:38.778873 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:38.778873 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:38.778873 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:38.778873 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:38.778873 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:38.778873 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:38.778873 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:38.778873 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:38.778873 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:38.778873 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:38.779794 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:38.779794 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:38.779794 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 20 08:48:38.779794 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 20 08:48:38.779794 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 20 08:48:38.779794 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 20 08:48:38.779794 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 20 08:48:38.779794 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 20 08:48:38.779794 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 20 08:48:38.779794 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:48:38.779794 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 247/494] 10.222.0.36 () {56 vars in 1096 bytes} [Thu Oct 20 08:48:38 2022] GET /identity/v3/users/swift => generated 83 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:38.784663 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a13c61-1608-4efc-98cb-ef16174c673c None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:38.914705 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a13c61-1608-4efc-98cb-ef16174c673c None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:38.944244 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:38.950291 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d093db1-2b3b-4084-a882-5ca8b06b42d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:38.950928 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d093db1-2b3b-4084-a882-5ca8b06b42d5 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:38.951204 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d093db1-2b3b-4084-a882-5ca8b06b42d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:38.951475 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d093db1-2b3b-4084-a882-5ca8b06b42d5 None admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:38.952259 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d093db1-2b3b-4084-a882-5ca8b06b42d5 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:38.952963 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d093db1-2b3b-4084-a882-5ca8b06b42d5 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:38.954789 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d093db1-2b3b-4084-a882-5ca8b06b42d5 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:38.973126 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 248/495] 10.222.0.36 () {56 vars in 1135 bytes} [Thu Oct 20 08:48:38 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:38.979070 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dc68a49-3a19-4b42-9f56-2e707b155b52 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:39.028227 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32a6ad3a-d588-4d1d-bf38-fbc88d8a8449 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:39.028566 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32a6ad3a-d588-4d1d-bf38-fbc88d8a8449 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:39.028844 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32a6ad3a-d588-4d1d-bf38-fbc88d8a8449 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:39.030309 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 249/496] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:48:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 20 08:48:39.035443 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c42593-046b-416a-8427-08aaeb0fb47e None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:39.035720 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c42593-046b-416a-8427-08aaeb0fb47e None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:39.035991 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c42593-046b-416a-8427-08aaeb0fb47e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:39.061833 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-f8c42593-046b-416a-8427-08aaeb0fb47e None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:48:39.063040 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-f8c42593-046b-416a-8427-08aaeb0fb47e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:48:39.164194 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dc68a49-3a19-4b42-9f56-2e707b155b52 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:39.175394 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:39.192278 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 250/497] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:48:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 20 08:48:39.209880 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb15a0d7-6ecc-44a9-a7ed-10ceb1fdb8b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:39.210510 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb15a0d7-6ecc-44a9-a7ed-10ceb1fdb8b7 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:39.210904 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb15a0d7-6ecc-44a9-a7ed-10ceb1fdb8b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:39.211232 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb15a0d7-6ecc-44a9-a7ed-10ceb1fdb8b7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:39.300620 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb15a0d7-6ecc-44a9-a7ed-10ceb1fdb8b7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:39.302743 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb15a0d7-6ecc-44a9-a7ed-10ceb1fdb8b7 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:39.407485 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 248/498] 10.222.0.36 () {58 vars in 1282 bytes} [Thu Oct 20 08:48:38 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 429 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 20 08:48:39.415580 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8c42593-046b-416a-8427-08aaeb0fb47e None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:39.483581 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb15a0d7-6ecc-44a9-a7ed-10ceb1fdb8b7 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:39.493278 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:39.571127 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8c42593-046b-416a-8427-08aaeb0fb47e None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:39.595145 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:39.614300 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e77093d4-c7ee-48ed-82df-8ac6490d2b16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:39.615912 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e77093d4-c7ee-48ed-82df-8ac6490d2b16 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:39.616479 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e77093d4-c7ee-48ed-82df-8ac6490d2b16 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:39.616955 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e77093d4-c7ee-48ed-82df-8ac6490d2b16 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:39.627786 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e77093d4-c7ee-48ed-82df-8ac6490d2b16 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:39.629543 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e77093d4-c7ee-48ed-82df-8ac6490d2b16 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:39.645034 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-e77093d4-c7ee-48ed-82df-8ac6490d2b16 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:39.645034 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:39.645034 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:39.645034 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:39.645034 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:39.645034 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:39.645034 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:39.645034 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:39.645034 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:39.645034 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:39.645034 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:39.645034 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:39.645034 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 20 08:48:39.645034 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 20 08:48:39.645034 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 20 08:48:39.645034 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 20 08:48:39.646074 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:39.646074 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:39.646074 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:39.646074 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:39.646074 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:39.646074 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:39.646074 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:39.646074 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:39.646074 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:39.646074 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:39.646074 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:39.646074 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:39.646074 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:39.646074 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:39.646074 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:39.646074 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:39.646074 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 20 08:48:39.646995 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 20 08:48:39.646995 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:39.646995 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:39.649087 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 251/499] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:48:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 234 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:39.655273 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e77093d4-c7ee-48ed-82df-8ac6490d2b16 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:39.662268 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb15a0d7-6ecc-44a9-a7ed-10ceb1fdb8b7 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:39.764178 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e77093d4-c7ee-48ed-82df-8ac6490d2b16 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:39.795205 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-294a276e-9d78-49b7-9882-f8094bdc9895 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:39.795853 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-294a276e-9d78-49b7-9882-f8094bdc9895 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:39.796132 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-294a276e-9d78-49b7-9882-f8094bdc9895 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:39.796438 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-294a276e-9d78-49b7-9882-f8094bdc9895 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:39.800590 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9afbd7a-b161-4393-a2e1-f30b6ce94a80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:39.801754 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9afbd7a-b161-4393-a2e1-f30b6ce94a80 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:39.802819 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9afbd7a-b161-4393-a2e1-f30b6ce94a80 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:39.803312 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9afbd7a-b161-4393-a2e1-f30b6ce94a80 None admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:39.804294 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9afbd7a-b161-4393-a2e1-f30b6ce94a80 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:39.804891 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9afbd7a-b161-4393-a2e1-f30b6ce94a80 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:39.806489 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9afbd7a-b161-4393-a2e1-f30b6ce94a80 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:39.820973 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 252/500] 10.222.0.36 () {56 vars in 1109 bytes} [Thu Oct 20 08:48:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:39.823529 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-294a276e-9d78-49b7-9882-f8094bdc9895 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 20 08:48:39.828853 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-294a276e-9d78-49b7-9882-f8094bdc9895 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 20 08:48:39.828853 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:39.828853 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:39.828853 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:39.828853 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:39.828853 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:39.828853 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:39.828853 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:39.828853 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:39.828853 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:39.828853 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:39.828853 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:39.828853 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 20 08:48:39.828853 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 20 08:48:39.828853 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 20 08:48:39.828853 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 20 08:48:39.829920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 20 08:48:39.829920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 20 08:48:39.829920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 20 08:48:39.829920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 20 08:48:39.829920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:39.829920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:39.829920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:39.829920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:39.829920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:39.829920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:39.829920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:39.829920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:39.829920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:39.829920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:39.829920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:39.829920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:39.831274 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:39.831274 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:39.831274 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:39.831274 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:39.831274 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 20 08:48:39.831274 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 20 08:48:39.831274 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 20 08:48:39.831274 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 20 08:48:39.831274 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 20 08:48:39.831274 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 20 08:48:39.831274 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 20 08:48:39.831274 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:48:39.833846 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9afbd7a-b161-4393-a2e1-f30b6ce94a80 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:39.844149 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 249/501] 10.222.0.36 () {56 vars in 1096 bytes} [Thu Oct 20 08:48:39 2022] GET /identity/v3/roles/admin => generated 83 bytes in 361 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:39.855511 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-294a276e-9d78-49b7-9882-f8094bdc9895 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:39.890924 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:39.944138 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9afbd7a-b161-4393-a2e1-f30b6ce94a80 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:39.979804 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70b5a0ab-3cca-42f4-bb3f-07d21a690056 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:39.980784 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70b5a0ab-3cca-42f4-bb3f-07d21a690056 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:39.981345 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70b5a0ab-3cca-42f4-bb3f-07d21a690056 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:39.981816 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70b5a0ab-3cca-42f4-bb3f-07d21a690056 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:39.989487 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70b5a0ab-3cca-42f4-bb3f-07d21a690056 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:39.991939 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70b5a0ab-3cca-42f4-bb3f-07d21a690056 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:40.003972 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-70b5a0ab-3cca-42f4-bb3f-07d21a690056 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 20 08:48:40.003972 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:40.003972 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:40.003972 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:40.003972 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:40.003972 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:40.003972 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:40.003972 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:40.003972 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:40.003972 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:40.003972 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:40.003972 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:40.003972 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 20 08:48:40.003972 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 20 08:48:40.003972 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 20 08:48:40.004894 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 20 08:48:40.004894 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:40.004894 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:40.004894 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:40.004894 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:40.004894 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:40.004894 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:40.004894 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:40.004894 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:40.004894 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:40.004894 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:40.004894 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:40.004894 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:40.004894 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:40.004894 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:40.004894 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:40.004894 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:40.006514 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 20 08:48:40.006514 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 20 08:48:40.006514 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 20 08:48:40.006514 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 20 08:48:40.006514 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 20 08:48:40.006514 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 20 08:48:40.006514 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 20 08:48:40.006514 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:40.008895 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 253/502] 10.222.0.36 () {56 vars in 1111 bytes} [Thu Oct 20 08:48:39 2022] GET /identity/v3/projects/service => generated 88 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:40.014029 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70b5a0ab-3cca-42f4-bb3f-07d21a690056 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:40.024981 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:40.093418 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-294a276e-9d78-49b7-9882-f8094bdc9895 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:40.171754 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45bfdfe1-3595-4f58-9f27-c939cd33d1c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:40.172722 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45bfdfe1-3595-4f58-9f27-c939cd33d1c8 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:40.173245 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45bfdfe1-3595-4f58-9f27-c939cd33d1c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:40.173714 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45bfdfe1-3595-4f58-9f27-c939cd33d1c8 None admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:40.174607 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45bfdfe1-3595-4f58-9f27-c939cd33d1c8 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:40.175108 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45bfdfe1-3595-4f58-9f27-c939cd33d1c8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:40.176673 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45bfdfe1-3595-4f58-9f27-c939cd33d1c8 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:40.196270 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 250/503] 10.222.0.36 () {56 vars in 1099 bytes} [Thu Oct 20 08:48:39 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 341 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:40.203894 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:40.209218 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45bfdfe1-3595-4f58-9f27-c939cd33d1c8 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:40.223659 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70b5a0ab-3cca-42f4-bb3f-07d21a690056 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:40.268703 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ef903d3-cd82-4361-802b-c0706a1e3957 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:40.269557 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef903d3-cd82-4361-802b-c0706a1e3957 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:40.270031 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef903d3-cd82-4361-802b-c0706a1e3957 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:40.270308 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef903d3-cd82-4361-802b-c0706a1e3957 None admin] PATH_INFO: `/v3/projects` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:40.271689 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ef903d3-cd82-4361-802b-c0706a1e3957 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:40.271991 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ef903d3-cd82-4361-802b-c0706a1e3957 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:40.273773 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ef903d3-cd82-4361-802b-c0706a1e3957 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:40.294570 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 254/504] 10.222.0.36 () {56 vars in 1148 bytes} [Thu Oct 20 08:48:40 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:40.294570 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f10ed2d-2f60-4b09-a6cf-ef53f40f592c None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:40.294570 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f10ed2d-2f60-4b09-a6cf-ef53f40f592c None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:40.294570 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f10ed2d-2f60-4b09-a6cf-ef53f40f592c None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:40.294570 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 255/505] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:48:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:48:40.294570 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f10ed2d-2f60-4b09-a6cf-ef53f40f592c None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:40.343901 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45bfdfe1-3595-4f58-9f27-c939cd33d1c8 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:40.375832 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:40.387463 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eec394e6-78bb-4144-8b8c-8a1c200fd69c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:40.388402 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec394e6-78bb-4144-8b8c-8a1c200fd69c None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:40.388941 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec394e6-78bb-4144-8b8c-8a1c200fd69c None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:40.389124 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec394e6-78bb-4144-8b8c-8a1c200fd69c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:40.400632 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eec394e6-78bb-4144-8b8c-8a1c200fd69c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:40.402083 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eec394e6-78bb-4144-8b8c-8a1c200fd69c None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:40.412988 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f10ed2d-2f60-4b09-a6cf-ef53f40f592c None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:40.416518 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-eec394e6-78bb-4144-8b8c-8a1c200fd69c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:40.416518 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:40.416518 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:40.416518 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:40.416518 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:40.416518 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:40.416518 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:40.416518 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:40.416518 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:40.416518 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:40.416518 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:40.416518 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:40.416518 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 20 08:48:40.416518 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 20 08:48:40.416518 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 20 08:48:40.416518 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 20 08:48:40.417432 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:40.417432 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:40.417432 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:40.417432 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:40.417432 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:40.417432 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:40.417432 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:40.417432 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:40.417432 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:40.417432 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:40.417432 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:40.417432 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:40.417432 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:40.417432 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:40.417432 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:40.417432 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:40.417432 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 20 08:48:40.423430 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 20 08:48:40.423430 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:40.423430 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:48:40.423430 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 251/506] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:48:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 20 08:48:40.423430 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c977e5-f3d9-43c1-b652-9ae79da38612 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:40.423430 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c977e5-f3d9-43c1-b652-9ae79da38612 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:40.423430 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c977e5-f3d9-43c1-b652-9ae79da38612 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:40.437647 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-c1c977e5-f3d9-43c1-b652-9ae79da38612 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:48:40.439036 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-c1c977e5-f3d9-43c1-b652-9ae79da38612 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:48:40.451786 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0250c93-f38e-4409-bbd3-54d53b99f38f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:40.452412 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0250c93-f38e-4409-bbd3-54d53b99f38f None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:40.452689 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0250c93-f38e-4409-bbd3-54d53b99f38f None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:40.452957 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0250c93-f38e-4409-bbd3-54d53b99f38f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:40.453671 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0250c93-f38e-4409-bbd3-54d53b99f38f None admin] RBAC: Adding query filter params (role.id=f81100d13381464aa48cbae3561ca1bc, scope.project.id=d9857b610fb542969cfb5efbbdb8ce61, user.id=4d558918cd8b4ebea4fe8765297304db) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:40.453976 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0250c93-f38e-4409-bbd3-54d53b99f38f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:40.455429 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0250c93-f38e-4409-bbd3-54d53b99f38f None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:40.462037 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 256/507] 10.222.0.36 () {56 vars in 1374 bytes} [Thu Oct 20 08:48:40 2022] GET /identity/v3/role_assignments?user.id=4d558918cd8b4ebea4fe8765297304db&scope.project.id=d9857b610fb542969cfb5efbbdb8ce61&role.id=f81100d13381464aa48cbae3561ca1bc => generated 608 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:40.463209 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-671a3b28-0550-4f3d-96f3-e6d6137df87d None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:40.463487 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-671a3b28-0550-4f3d-96f3-e6d6137df87d None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:40.463758 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-671a3b28-0550-4f3d-96f3-e6d6137df87d None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:40.466512 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 257/508] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:48:40 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:48:40.466512 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-671a3b28-0550-4f3d-96f3-e6d6137df87d None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:40.486963 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:40.524672 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 252/509] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:48:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 20 08:48:40.526547 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6802ac-b257-41c0-a2af-8df8a263e8ed None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:40.527051 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6802ac-b257-41c0-a2af-8df8a263e8ed None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:40.527529 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6802ac-b257-41c0-a2af-8df8a263e8ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:40.548026 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-1c6802ac-b257-41c0-a2af-8df8a263e8ed None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:48:40.549615 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-1c6802ac-b257-41c0-a2af-8df8a263e8ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:48:40.615050 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-671a3b28-0550-4f3d-96f3-e6d6137df87d None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:40.683535 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 253/510] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:48:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 20 08:48:40.684323 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a65e66f-67ce-4e90-98ef-bcc965f250c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:40.684995 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a65e66f-67ce-4e90-98ef-bcc965f250c7 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:40.685271 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a65e66f-67ce-4e90-98ef-bcc965f250c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:40.685538 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a65e66f-67ce-4e90-98ef-bcc965f250c7 None admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:40.686187 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a65e66f-67ce-4e90-98ef-bcc965f250c7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:40.686495 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a65e66f-67ce-4e90-98ef-bcc965f250c7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:40.687923 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a65e66f-67ce-4e90-98ef-bcc965f250c7 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:40.709371 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 258/511] 10.222.0.36 () {56 vars in 1109 bytes} [Thu Oct 20 08:48:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:40.714911 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c6802ac-b257-41c0-a2af-8df8a263e8ed None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:40.763938 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:40.827425 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c6802ac-b257-41c0-a2af-8df8a263e8ed None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:40.860867 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d1b48a0-db81-49f3-b1a2-ed3a219fb9de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:40.861646 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1b48a0-db81-49f3-b1a2-ed3a219fb9de None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:40.861922 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1b48a0-db81-49f3-b1a2-ed3a219fb9de None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:40.862186 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1b48a0-db81-49f3-b1a2-ed3a219fb9de None admin] PATH_INFO: `/v3/users/nova` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:40.862530 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a65e66f-67ce-4e90-98ef-bcc965f250c7 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:40.870840 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d1b48a0-db81-49f3-b1a2-ed3a219fb9de None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:40.872298 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d1b48a0-db81-49f3-b1a2-ed3a219fb9de None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:40.888957 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-5d1b48a0-db81-49f3-b1a2-ed3a219fb9de None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 20 08:48:40.888957 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:40.888957 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:40.888957 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:40.888957 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:40.888957 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:40.888957 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:40.888957 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:40.888957 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:40.888957 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:40.888957 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:40.888957 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:40.888957 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 20 08:48:40.888957 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 20 08:48:40.888957 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 20 08:48:40.888957 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 20 08:48:40.889957 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:40.889957 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:40.889957 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 20 08:48:40.889957 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 20 08:48:40.889957 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 20 08:48:40.889957 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 20 08:48:40.889957 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:40.889957 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:40.889957 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:40.889957 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:40.889957 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:40.889957 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:40.889957 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:40.889957 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:40.889957 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:40.889957 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:40.889957 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:40.889957 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:40.890958 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:40.890958 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:40.890958 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 20 08:48:40.890958 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 20 08:48:40.890958 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 20 08:48:40.890958 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 20 08:48:40.890958 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 20 08:48:40.890958 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 20 08:48:40.890958 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 20 08:48:40.890958 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:48:40.890958 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 254/512] 10.222.0.36 () {56 vars in 1093 bytes} [Thu Oct 20 08:48:40 2022] GET /identity/v3/users/nova => generated 82 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:40.897800 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d1b48a0-db81-49f3-b1a2-ed3a219fb9de None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:40.963241 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:40.994397 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a65e66f-67ce-4e90-98ef-bcc965f250c7 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:41.033816 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:41.037023 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb311bca-4ae3-4189-b129-df5c15d408e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:41.037957 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb311bca-4ae3-4189-b129-df5c15d408e6 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:41.038282 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb311bca-4ae3-4189-b129-df5c15d408e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:41.038417 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d1b48a0-db81-49f3-b1a2-ed3a219fb9de None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:41.038618 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb311bca-4ae3-4189-b129-df5c15d408e6 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:41.039223 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb311bca-4ae3-4189-b129-df5c15d408e6 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:41.041482 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb311bca-4ae3-4189-b129-df5c15d408e6 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:41.055230 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-cb311bca-4ae3-4189-b129-df5c15d408e6 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 20 08:48:41.055230 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:41.055230 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 20 08:48:41.055230 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 20 08:48:41.055230 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 20 08:48:41.055230 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 20 08:48:41.055230 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 20 08:48:41.055230 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 20 08:48:41.055230 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 20 08:48:41.055230 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:41.055230 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 20 08:48:41.055230 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:41.055230 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:41.055230 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:41.055230 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:41.055230 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:41.055230 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:41.056306 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:41.056306 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:41.056306 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:41.056306 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:41.056306 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:41.056306 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:41.056306 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 20 08:48:41.056306 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 20 08:48:41.056306 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 20 08:48:41.056306 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 20 08:48:41.056306 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:41.056306 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:41.056306 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:41.056306 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:41.056306 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:41.056306 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:41.057154 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:41.057154 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:41.057154 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:41.057154 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:41.057154 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:41.057154 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:41.057154 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:41.057154 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:41.057154 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:41.057154 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:41.057154 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 20 08:48:41.057154 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 20 08:48:41.057154 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 20 08:48:41.057154 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:41.057901 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 259/513] 10.222.0.36 () {56 vars in 1129 bytes} [Thu Oct 20 08:48:40 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:41.062518 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb311bca-4ae3-4189-b129-df5c15d408e6 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:41.084101 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aa34b28-9cb7-4e34-897f-bdd1e2695f3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:41.084624 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa34b28-9cb7-4e34-897f-bdd1e2695f3c None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:41.084897 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa34b28-9cb7-4e34-897f-bdd1e2695f3c None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:41.085163 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa34b28-9cb7-4e34-897f-bdd1e2695f3c None admin] PATH_INFO: `/v3/users` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:41.085889 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aa34b28-9cb7-4e34-897f-bdd1e2695f3c None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:41.086186 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aa34b28-9cb7-4e34-897f-bdd1e2695f3c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:41.087665 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aa34b28-9cb7-4e34-897f-bdd1e2695f3c None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:41.109453 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 255/514] 10.222.0.36 () {56 vars in 1133 bytes} [Thu Oct 20 08:48:40 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:41.110954 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aa34b28-9cb7-4e34-897f-bdd1e2695f3c None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:41.179496 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb311bca-4ae3-4189-b129-df5c15d408e6 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:41.214373 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e4652e4-3837-4fcc-a1a2-a193c030978c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:41.215036 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e4652e4-3837-4fcc-a1a2-a193c030978c None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:41.215404 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e4652e4-3837-4fcc-a1a2-a193c030978c None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:41.215668 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e4652e4-3837-4fcc-a1a2-a193c030978c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:41.218728 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aa34b28-9cb7-4e34-897f-bdd1e2695f3c None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:41.223511 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:41.227979 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e4652e4-3837-4fcc-a1a2-a193c030978c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:41.230181 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e4652e4-3837-4fcc-a1a2-a193c030978c None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:41.246617 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-9e4652e4-3837-4fcc-a1a2-a193c030978c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:41.246617 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:41.246617 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:41.246617 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:41.246617 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:41.246617 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:41.246617 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:41.246617 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:41.246617 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:41.246617 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:41.246617 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:41.246617 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:41.246617 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 20 08:48:41.246617 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 20 08:48:41.246617 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 20 08:48:41.246617 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 20 08:48:41.247572 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:41.247572 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:41.247572 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:41.247572 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:41.247572 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:41.247572 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:41.247572 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:41.247572 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:41.247572 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:41.247572 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:41.247572 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:41.247572 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:41.247572 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:41.247572 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:41.247572 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:41.247572 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:41.247572 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 20 08:48:41.248535 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 20 08:48:41.248535 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:41.248535 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:41.248535 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 260/515] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:48:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:41.248535 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e4652e4-3837-4fcc-a1a2-a193c030978c None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:41.256976 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-106ca942-b438-4327-b9a4-87f555ebcd13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:41.257580 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-106ca942-b438-4327-b9a4-87f555ebcd13 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:41.257853 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-106ca942-b438-4327-b9a4-87f555ebcd13 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:41.258116 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-106ca942-b438-4327-b9a4-87f555ebcd13 None admin] PATH_INFO: `/v3/services` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:41.258771 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-106ca942-b438-4327-b9a4-87f555ebcd13 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:41.259068 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-106ca942-b438-4327-b9a4-87f555ebcd13 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:41.260435 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-106ca942-b438-4327-b9a4-87f555ebcd13 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:41.269260 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 256/516] 10.222.0.36 () {56 vars in 1124 bytes} [Thu Oct 20 08:48:41 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:41.269260 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82280c14-f93e-42eb-9666-db9c9ac8e24e None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:41.269260 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82280c14-f93e-42eb-9666-db9c9ac8e24e None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:41.269260 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82280c14-f93e-42eb-9666-db9c9ac8e24e None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:41.269260 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 257/517] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:48:41 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:48:41.269260 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82280c14-f93e-42eb-9666-db9c9ac8e24e None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:41.297321 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:41.371493 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e4652e4-3837-4fcc-a1a2-a193c030978c None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:41.379167 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82280c14-f93e-42eb-9666-db9c9ac8e24e None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:41.413712 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee4c33a5-13dc-4b57-b098-3ae1ab7dcd5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:41.414204 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0890980-88e8-46ab-aed5-4ec9a128bf86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:41.414340 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee4c33a5-13dc-4b57-b098-3ae1ab7dcd5a None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:41.414632 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee4c33a5-13dc-4b57-b098-3ae1ab7dcd5a None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:41.414817 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0890980-88e8-46ab-aed5-4ec9a128bf86 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:41.414914 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee4c33a5-13dc-4b57-b098-3ae1ab7dcd5a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:41.415121 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0890980-88e8-46ab-aed5-4ec9a128bf86 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:41.415408 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0890980-88e8-46ab-aed5-4ec9a128bf86 None admin] PATH_INFO: `/v3/domains` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:41.415973 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0890980-88e8-46ab-aed5-4ec9a128bf86 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:41.416269 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0890980-88e8-46ab-aed5-4ec9a128bf86 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:41.417873 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0890980-88e8-46ab-aed5-4ec9a128bf86 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:41.425327 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee4c33a5-13dc-4b57-b098-3ae1ab7dcd5a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:41.426786 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee4c33a5-13dc-4b57-b098-3ae1ab7dcd5a None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:41.434162 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 258/518] 10.222.0.36 () {56 vars in 1109 bytes} [Thu Oct 20 08:48:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:41.434162 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0890980-88e8-46ab-aed5-4ec9a128bf86 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:41.443760 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-ee4c33a5-13dc-4b57-b098-3ae1ab7dcd5a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:41.443760 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:41.443760 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:41.443760 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:41.443760 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:41.443760 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:41.443760 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:41.443760 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:41.443760 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:41.443760 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:41.443760 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:41.443760 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:41.443760 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 20 08:48:41.443760 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 20 08:48:41.443760 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 20 08:48:41.443760 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 20 08:48:41.444700 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:41.444700 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:41.444700 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:41.444700 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:41.444700 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:41.444700 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:41.444700 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:41.444700 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:41.444700 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:41.444700 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:41.444700 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:41.444700 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:41.444700 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:41.444700 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:41.444700 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:41.444700 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:41.444700 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 20 08:48:41.450319 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 20 08:48:41.450319 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:41.450319 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:41.450319 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 261/519] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:48:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:41.450319 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5abf3a7-ec2b-4b78-a265-cd0b90ca87ae None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:41.450319 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5abf3a7-ec2b-4b78-a265-cd0b90ca87ae None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:41.450319 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5abf3a7-ec2b-4b78-a265-cd0b90ca87ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:41.467480 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-e5abf3a7-ec2b-4b78-a265-cd0b90ca87ae None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:48:41.468684 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-e5abf3a7-ec2b-4b78-a265-cd0b90ca87ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:48:41.506609 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:41.525283 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:41.543739 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 262/520] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:48:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 20 08:48:41.544696 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5abf3a7-ec2b-4b78-a265-cd0b90ca87ae None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:41.545171 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0890980-88e8-46ab-aed5-4ec9a128bf86 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:41.594798 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2161b00-bda1-492a-8a76-06e0570d78ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:41.595406 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2161b00-bda1-492a-8a76-06e0570d78ea None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:41.595681 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2161b00-bda1-492a-8a76-06e0570d78ea None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:41.595948 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2161b00-bda1-492a-8a76-06e0570d78ea None admin] PATH_INFO: `/v3/services` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:41.596575 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2161b00-bda1-492a-8a76-06e0570d78ea None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:41.596870 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2161b00-bda1-492a-8a76-06e0570d78ea None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:41.598820 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2161b00-bda1-492a-8a76-06e0570d78ea None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:41.614497 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 259/521] 10.222.0.36 () {56 vars in 1124 bytes} [Thu Oct 20 08:48:41 2022] GET /identity/v3/services?type=block-storage => generated 130 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:41.614497 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2161b00-bda1-492a-8a76-06e0570d78ea None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:41.687250 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5abf3a7-ec2b-4b78-a265-cd0b90ca87ae None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:41.727787 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-606bfc81-1fe9-4ca2-892f-a4abf8741f8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:41.728815 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606bfc81-1fe9-4ca2-892f-a4abf8741f8d None admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:41.729347 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606bfc81-1fe9-4ca2-892f-a4abf8741f8d None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:41.729813 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606bfc81-1fe9-4ca2-892f-a4abf8741f8d None admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:41.730697 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-606bfc81-1fe9-4ca2-892f-a4abf8741f8d None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:41.732363 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-606bfc81-1fe9-4ca2-892f-a4abf8741f8d None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:41.764643 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 263/522] 10.222.0.36 () {60 vars in 1132 bytes} [Thu Oct 20 08:48:41 2022] POST /identity/v3/users => generated 254 bytes in 220 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 08:48:41.842629 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2161b00-bda1-492a-8a76-06e0570d78ea None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:41.881895 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bc53c28-c160-4970-8082-2cd22ee11d49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:41.882937 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc53c28-c160-4970-8082-2cd22ee11d49 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:41.883486 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc53c28-c160-4970-8082-2cd22ee11d49 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:41.883959 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc53c28-c160-4970-8082-2cd22ee11d49 None admin] PATH_INFO: `/v3/domains` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:41.884819 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bc53c28-c160-4970-8082-2cd22ee11d49 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:41.885414 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bc53c28-c160-4970-8082-2cd22ee11d49 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:41.887150 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bc53c28-c160-4970-8082-2cd22ee11d49 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:41.902346 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 260/523] 10.222.0.36 () {56 vars in 1109 bytes} [Thu Oct 20 08:48:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 294 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:41.904054 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-606bfc81-1fe9-4ca2-892f-a4abf8741f8d None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:41.909933 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bc53c28-c160-4970-8082-2cd22ee11d49 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:41.920620 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:41.940314 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:42.020638 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-606bfc81-1fe9-4ca2-892f-a4abf8741f8d None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:42.059279 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bc53c28-c160-4970-8082-2cd22ee11d49 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:42.059809 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c8a3eaa-ee17-400a-ae79-b4b379023167 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:42.060785 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c8a3eaa-ee17-400a-ae79-b4b379023167 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:42.061258 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c8a3eaa-ee17-400a-ae79-b4b379023167 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:42.061792 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c8a3eaa-ee17-400a-ae79-b4b379023167 None admin] PATH_INFO: `/v3/services/network` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:42.062594 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c8a3eaa-ee17-400a-ae79-b4b379023167 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:42.064150 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c8a3eaa-ee17-400a-ae79-b4b379023167 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:42.074215 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-0c8a3eaa-ee17-400a-ae79-b4b379023167 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Oct 20 08:48:42.074215 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:42.074215 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 20 08:48:42.074215 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 20 08:48:42.074215 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 20 08:48:42.074215 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 20 08:48:42.074215 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 20 08:48:42.074215 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 20 08:48:42.074215 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 20 08:48:42.074215 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:42.074215 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 20 08:48:42.074215 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:42.074215 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:42.074215 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:42.074215 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:42.074215 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:42.074215 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:42.075298 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:42.075298 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:42.075298 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:42.075298 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:42.075298 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:42.075298 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:42.075298 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 20 08:48:42.075298 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 20 08:48:42.075298 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 20 08:48:42.075298 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 20 08:48:42.075298 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:42.075298 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:42.075298 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:42.075298 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:42.075298 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:42.075298 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:42.076246 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:42.076246 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:42.076246 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:42.076246 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:42.076246 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:42.076246 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:42.076246 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:42.076246 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:42.076246 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:42.076246 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:42.076246 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 20 08:48:42.076246 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 20 08:48:42.076246 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 20 08:48:42.076246 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:42.084235 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 264/524] 10.222.0.36 () {56 vars in 1111 bytes} [Thu Oct 20 08:48:41 2022] GET /identity/v3/services/network => generated 88 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 20 08:48:42.085847 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c8a3eaa-ee17-400a-ae79-b4b379023167 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:42.100113 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcfba899-db31-4656-8520-5b92b386454d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:42.100747 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcfba899-db31-4656-8520-5b92b386454d None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:42.101025 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcfba899-db31-4656-8520-5b92b386454d None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:42.101295 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcfba899-db31-4656-8520-5b92b386454d None admin] PATH_INFO: `/v3/projects/service` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:42.108917 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcfba899-db31-4656-8520-5b92b386454d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:42.110395 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcfba899-db31-4656-8520-5b92b386454d None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:42.123017 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-fcfba899-db31-4656-8520-5b92b386454d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 20 08:48:42.123017 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:42.123017 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:42.123017 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:42.123017 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:42.123017 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:42.123017 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:42.123017 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:42.123017 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:42.123017 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:42.123017 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:42.123017 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:42.123017 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 20 08:48:42.123017 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 20 08:48:42.123017 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 20 08:48:42.124038 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 20 08:48:42.124038 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:42.124038 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:42.124038 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:42.124038 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:42.124038 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:42.124038 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:42.124038 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:42.124038 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:42.124038 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:42.124038 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:42.124038 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:42.124038 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:42.124038 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:42.124038 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:42.124038 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:42.124038 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:42.124930 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 20 08:48:42.124930 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 20 08:48:42.124930 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 20 08:48:42.124930 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 20 08:48:42.124930 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 20 08:48:42.124930 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 20 08:48:42.124930 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 20 08:48:42.124930 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:48:42.129887 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 261/525] 10.222.0.36 () {56 vars in 1111 bytes} [Thu Oct 20 08:48:41 2022] GET /identity/v3/projects/service => generated 88 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:42.130860 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcfba899-db31-4656-8520-5b92b386454d None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:42.190555 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:42.200031 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c8a3eaa-ee17-400a-ae79-b4b379023167 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:42.209757 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:42.239217 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e740a63b-11dd-4621-8392-4c7e10798547 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:42.240266 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e740a63b-11dd-4621-8392-4c7e10798547 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:42.240789 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e740a63b-11dd-4621-8392-4c7e10798547 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:42.241276 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e740a63b-11dd-4621-8392-4c7e10798547 None admin] PATH_INFO: `/v3/services` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:42.242108 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e740a63b-11dd-4621-8392-4c7e10798547 None admin] RBAC: Adding query filter params (name=network) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:42.242775 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e740a63b-11dd-4621-8392-4c7e10798547 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:42.244452 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e740a63b-11dd-4621-8392-4c7e10798547 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:42.249381 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcfba899-db31-4656-8520-5b92b386454d None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:42.254120 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 265/526] 10.222.0.36 () {56 vars in 1112 bytes} [Thu Oct 20 08:48:42 2022] GET /identity/v3/services?name=network => generated 124 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 08:48:42.255649 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e740a63b-11dd-4621-8392-4c7e10798547 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:42.285091 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edeeab93-11d4-4171-8f8e-73fa3d5885d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:42.285708 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edeeab93-11d4-4171-8f8e-73fa3d5885d3 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:42.285987 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edeeab93-11d4-4171-8f8e-73fa3d5885d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:42.286259 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edeeab93-11d4-4171-8f8e-73fa3d5885d3 None admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:42.287771 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edeeab93-11d4-4171-8f8e-73fa3d5885d3 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:42.288081 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edeeab93-11d4-4171-8f8e-73fa3d5885d3 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:42.289522 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edeeab93-11d4-4171-8f8e-73fa3d5885d3 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:42.301974 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 262/527] 10.222.0.36 () {56 vars in 1148 bytes} [Thu Oct 20 08:48:42 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:42.307753 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edeeab93-11d4-4171-8f8e-73fa3d5885d3 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:42.363628 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e740a63b-11dd-4621-8392-4c7e10798547 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:42.402728 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8084cc40-117e-458d-9030-a77ccc212d2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:42.403799 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8084cc40-117e-458d-9030-a77ccc212d2f None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:42.408460 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8084cc40-117e-458d-9030-a77ccc212d2f None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:42.408956 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8084cc40-117e-458d-9030-a77ccc212d2f None admin] PATH_INFO: `/v3/services` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:42.409783 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8084cc40-117e-458d-9030-a77ccc212d2f None admin] RBAC: Adding query filter params (type=network) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:42.410289 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8084cc40-117e-458d-9030-a77ccc212d2f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:42.412465 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8084cc40-117e-458d-9030-a77ccc212d2f None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:42.423896 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 266/528] 10.222.0.36 () {56 vars in 1112 bytes} [Thu Oct 20 08:48:42 2022] GET /identity/v3/services?type=network => generated 124 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 08:48:42.462780 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edeeab93-11d4-4171-8f8e-73fa3d5885d3 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:42.509038 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:42.518997 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48afbc14-e558-4cdb-affc-42312bf7982b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:42.519933 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48afbc14-e558-4cdb-affc-42312bf7982b None admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:42.520218 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48afbc14-e558-4cdb-affc-42312bf7982b None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:42.520495 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48afbc14-e558-4cdb-affc-42312bf7982b None admin] PATH_INFO: `/v3/projects/d9857b610fb542969cfb5efbbdb8ce61/users/a94e868115ec49918eda10724263930c/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:42.522587 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48afbc14-e558-4cdb-affc-42312bf7982b None admin] RBAC: Authorizing `identity:create_grant(project_id=d9857b610fb542969cfb5efbbdb8ce61, user_id=a94e868115ec49918eda10724263930c, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:42.524083 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48afbc14-e558-4cdb-affc-42312bf7982b None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:42.549105 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 263/529] 10.222.0.36 () {58 vars in 1439 bytes} [Thu Oct 20 08:48:42 2022] PUT /identity/v3/projects/d9857b610fb542969cfb5efbbdb8ce61/users/a94e868115ec49918eda10724263930c/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 241 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 08:48:43.165546 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-317c2ffa-2160-4ae9-ba95-d6355e210dc8 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:43.166619 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-317c2ffa-2160-4ae9-ba95-d6355e210dc8 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:43.167099 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-317c2ffa-2160-4ae9-ba95-d6355e210dc8 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:43.168934 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 267/530] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:48:43 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:48:43.174352 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c553f99-444c-41a2-b115-79bc925e73e1 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:43.175182 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c553f99-444c-41a2-b115-79bc925e73e1 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:43.175663 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c553f99-444c-41a2-b115-79bc925e73e1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:43.203483 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-4c553f99-444c-41a2-b115-79bc925e73e1 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:48:43.205015 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-4c553f99-444c-41a2-b115-79bc925e73e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:48:43.284777 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 264/531] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:48:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 20 08:48:43.674792 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-317c2ffa-2160-4ae9-ba95-d6355e210dc8 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:43.698414 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:43.812141 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-317c2ffa-2160-4ae9-ba95-d6355e210dc8 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:43.857034 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-090df5b2-4ca6-4b48-821a-92e89c8272a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:43.857660 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090df5b2-4ca6-4b48-821a-92e89c8272a8 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:43.857934 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090df5b2-4ca6-4b48-821a-92e89c8272a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:43.858202 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090df5b2-4ca6-4b48-821a-92e89c8272a8 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:43.872968 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-090df5b2-4ca6-4b48-821a-92e89c8272a8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 20 08:48:43.877699 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-090df5b2-4ca6-4b48-821a-92e89c8272a8 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 20 08:48:43.877699 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:43.877699 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:43.877699 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:43.877699 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:43.877699 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:43.877699 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:43.877699 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:43.877699 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:43.877699 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:43.877699 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:43.877699 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:43.877699 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 20 08:48:43.877699 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 20 08:48:43.877699 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 20 08:48:43.877699 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 20 08:48:43.878651 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 20 08:48:43.878651 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 20 08:48:43.878651 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 20 08:48:43.878651 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 20 08:48:43.878651 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:43.878651 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:43.878651 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:43.878651 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:43.878651 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:43.878651 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:43.878651 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:43.878651 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:43.878651 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:43.878651 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:43.878651 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:43.878651 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:43.879901 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:43.879901 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:43.879901 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:43.879901 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:43.879901 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 20 08:48:43.879901 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 20 08:48:43.879901 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 20 08:48:43.879901 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 20 08:48:43.879901 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 20 08:48:43.879901 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 20 08:48:43.879901 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 20 08:48:43.879901 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:43.879901 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 268/532] 10.222.0.36 () {56 vars in 1096 bytes} [Thu Oct 20 08:48:43 2022] GET /identity/v3/roles/admin => generated 83 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:43.885443 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c553f99-444c-41a2-b115-79bc925e73e1 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:44.022226 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c553f99-444c-41a2-b115-79bc925e73e1 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:44.046147 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:44.068828 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a393abf0-00e4-47ca-8bbb-b9bb5f5a7bce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:44.069443 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a393abf0-00e4-47ca-8bbb-b9bb5f5a7bce None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:44.069719 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a393abf0-00e4-47ca-8bbb-b9bb5f5a7bce None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:44.069984 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a393abf0-00e4-47ca-8bbb-b9bb5f5a7bce None admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:44.070684 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a393abf0-00e4-47ca-8bbb-b9bb5f5a7bce None admin] RBAC: Adding query filter params (name=admin) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:44.070982 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a393abf0-00e4-47ca-8bbb-b9bb5f5a7bce None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:44.072376 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a393abf0-00e4-47ca-8bbb-b9bb5f5a7bce None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:44.086790 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 265/533] 10.222.0.36 () {56 vars in 1099 bytes} [Thu Oct 20 08:48:43 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:44.092244 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-090df5b2-4ca6-4b48-821a-92e89c8272a8 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:44.198532 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:44.225326 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-090df5b2-4ca6-4b48-821a-92e89c8272a8 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:44.271850 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c84b35ea-b3d8-47f2-9fd8-221001bbecd7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:44.272575 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c84b35ea-b3d8-47f2-9fd8-221001bbecd7 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:44.272809 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c84b35ea-b3d8-47f2-9fd8-221001bbecd7 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:44.273078 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c84b35ea-b3d8-47f2-9fd8-221001bbecd7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:44.284014 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c84b35ea-b3d8-47f2-9fd8-221001bbecd7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:44.285486 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c84b35ea-b3d8-47f2-9fd8-221001bbecd7 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:44.300066 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-c84b35ea-b3d8-47f2-9fd8-221001bbecd7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:44.300066 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:44.300066 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:44.300066 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:44.300066 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:44.300066 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:44.300066 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:44.300066 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:44.300066 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:44.300066 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:44.300066 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:44.300066 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:44.300066 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 20 08:48:44.300066 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 20 08:48:44.300066 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 20 08:48:44.300066 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 20 08:48:44.301161 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:44.301161 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:44.301161 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:44.301161 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:44.301161 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:44.301161 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:44.301161 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:44.301161 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:44.301161 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:44.301161 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:44.301161 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:44.301161 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:44.301161 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:44.301161 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:44.301161 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:44.301161 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:44.301161 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 20 08:48:44.303095 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 20 08:48:44.303095 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:44.303095 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:44.303095 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 269/534] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:48:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:44.307598 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a393abf0-00e4-47ca-8bbb-b9bb5f5a7bce None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:44.440679 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a393abf0-00e4-47ca-8bbb-b9bb5f5a7bce None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:44.478509 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438326d1-9fe0-4d78-8ebe-36eba1b8be79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:44.479194 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438326d1-9fe0-4d78-8ebe-36eba1b8be79 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:44.479471 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438326d1-9fe0-4d78-8ebe-36eba1b8be79 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:44.479750 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438326d1-9fe0-4d78-8ebe-36eba1b8be79 None admin] PATH_INFO: `/v3/domains` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:44.480450 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-438326d1-9fe0-4d78-8ebe-36eba1b8be79 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:44.480752 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-438326d1-9fe0-4d78-8ebe-36eba1b8be79 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:44.482170 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-438326d1-9fe0-4d78-8ebe-36eba1b8be79 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:44.500215 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 266/535] 10.222.0.36 () {56 vars in 1109 bytes} [Thu Oct 20 08:48:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:44.505931 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c84b35ea-b3d8-47f2-9fd8-221001bbecd7 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:44.622766 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c84b35ea-b3d8-47f2-9fd8-221001bbecd7 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:44.630992 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a77c3ab-e535-4981-abbf-33501a8fcf8c None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:44.631282 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a77c3ab-e535-4981-abbf-33501a8fcf8c None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:44.631555 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a77c3ab-e535-4981-abbf-33501a8fcf8c None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:44.632891 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 267/536] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:48:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 20 08:48:44.637883 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-800da4e5-e8bf-41fa-aeb6-7f9958d53c84 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:44.638166 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-800da4e5-e8bf-41fa-aeb6-7f9958d53c84 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:44.638443 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-800da4e5-e8bf-41fa-aeb6-7f9958d53c84 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:44.657314 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-800da4e5-e8bf-41fa-aeb6-7f9958d53c84 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:48:44.658666 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-800da4e5-e8bf-41fa-aeb6-7f9958d53c84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:48:44.663941 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:44.669817 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fd417dc-00f0-4709-9348-24572285b63c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:44.670898 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd417dc-00f0-4709-9348-24572285b63c None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:44.671323 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd417dc-00f0-4709-9348-24572285b63c None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:44.671785 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd417dc-00f0-4709-9348-24572285b63c None admin] PATH_INFO: `/v3/users/swift` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:44.681205 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fd417dc-00f0-4709-9348-24572285b63c None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:44.683258 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fd417dc-00f0-4709-9348-24572285b63c None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:44.695883 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:44.698361 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-2fd417dc-00f0-4709-9348-24572285b63c None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 20 08:48:44.698361 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:44.698361 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:44.698361 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:44.698361 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:44.698361 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:44.698361 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:44.698361 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:44.698361 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:44.698361 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:44.698361 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:44.698361 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:44.698361 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 20 08:48:44.698361 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 20 08:48:44.698361 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 20 08:48:44.698361 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 20 08:48:44.699514 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:44.699514 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:44.699514 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 20 08:48:44.699514 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 20 08:48:44.699514 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 20 08:48:44.699514 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 20 08:48:44.699514 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:44.699514 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:44.699514 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:44.699514 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:44.699514 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:44.699514 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:44.699514 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:44.699514 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:44.699514 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:44.699514 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:44.699514 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:44.699514 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:44.700648 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:44.700648 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:44.700648 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 20 08:48:44.700648 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 20 08:48:44.700648 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 20 08:48:44.700648 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 20 08:48:44.700648 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 20 08:48:44.700648 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 20 08:48:44.700648 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 20 08:48:44.700648 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:44.703729 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 270/537] 10.222.0.36 () {56 vars in 1096 bytes} [Thu Oct 20 08:48:44 2022] GET /identity/v3/users/swift => generated 83 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:44.709771 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b54eab7d-afb7-41c4-af1c-ded7e9708bf5 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:44.710706 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b54eab7d-afb7-41c4-af1c-ded7e9708bf5 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:44.711141 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b54eab7d-afb7-41c4-af1c-ded7e9708bf5 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:44.713213 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 271/538] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:48:44 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:48:44.714561 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b54eab7d-afb7-41c4-af1c-ded7e9708bf5 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:44.741554 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 268/539] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:48:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 20 08:48:44.743120 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71a04a1-ebdd-4bce-aacf-1aec5006f244 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:44.743392 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71a04a1-ebdd-4bce-aacf-1aec5006f244 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:44.743670 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71a04a1-ebdd-4bce-aacf-1aec5006f244 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:44.763456 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-c71a04a1-ebdd-4bce-aacf-1aec5006f244 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:48:44.764643 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-c71a04a1-ebdd-4bce-aacf-1aec5006f244 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:48:44.844192 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b54eab7d-afb7-41c4-af1c-ded7e9708bf5 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:44.856710 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 269/540] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:48:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 20 08:48:44.880871 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7641396a-5cbf-48d9-9f66-5f48aee2d716 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:44.881771 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7641396a-5cbf-48d9-9f66-5f48aee2d716 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:44.882243 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7641396a-5cbf-48d9-9f66-5f48aee2d716 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:44.882746 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7641396a-5cbf-48d9-9f66-5f48aee2d716 None admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:44.883645 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7641396a-5cbf-48d9-9f66-5f48aee2d716 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:44.884125 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7641396a-5cbf-48d9-9f66-5f48aee2d716 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:44.885738 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7641396a-5cbf-48d9-9f66-5f48aee2d716 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:44.904479 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 272/541] 10.222.0.36 () {56 vars in 1135 bytes} [Thu Oct 20 08:48:44 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:44.910392 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c71a04a1-ebdd-4bce-aacf-1aec5006f244 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:45.013644 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:45.017826 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c71a04a1-ebdd-4bce-aacf-1aec5006f244 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:45.055180 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3dae591-df9c-4867-97de-c27cd234befe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:45.055803 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3dae591-df9c-4867-97de-c27cd234befe None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:45.056074 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3dae591-df9c-4867-97de-c27cd234befe None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:45.056383 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3dae591-df9c-4867-97de-c27cd234befe None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:45.066666 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3dae591-df9c-4867-97de-c27cd234befe None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:45.068742 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3dae591-df9c-4867-97de-c27cd234befe None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:45.085305 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-d3dae591-df9c-4867-97de-c27cd234befe None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:45.085305 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:45.085305 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:45.085305 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:45.085305 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:45.085305 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:45.085305 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:45.085305 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:45.085305 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:45.085305 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:45.085305 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:45.085305 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:45.085305 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 20 08:48:45.085305 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 20 08:48:45.085305 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 20 08:48:45.085305 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 20 08:48:45.086264 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:45.086264 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:45.086264 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:45.086264 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:45.086264 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:45.086264 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:45.086264 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:45.086264 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:45.086264 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:45.086264 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:45.086264 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:45.086264 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:45.086264 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:45.086264 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:45.086264 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:45.086264 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:45.086264 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 20 08:48:45.087202 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 20 08:48:45.087202 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:45.087202 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:48:45.087202 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 270/542] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:48:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:45.093468 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7641396a-5cbf-48d9-9f66-5f48aee2d716 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:45.183299 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3dae591-df9c-4867-97de-c27cd234befe None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:45.183492 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:45.229448 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7641396a-5cbf-48d9-9f66-5f48aee2d716 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:45.270122 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d1592a4-b025-472d-a99c-46a7bfbae27b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:45.272018 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1592a4-b025-472d-a99c-46a7bfbae27b None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:45.272524 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1592a4-b025-472d-a99c-46a7bfbae27b None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:45.273011 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1592a4-b025-472d-a99c-46a7bfbae27b None admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:45.273849 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d1592a4-b025-472d-a99c-46a7bfbae27b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:45.274795 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d1592a4-b025-472d-a99c-46a7bfbae27b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:45.276848 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d1592a4-b025-472d-a99c-46a7bfbae27b None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:45.291938 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 273/543] 10.222.0.36 () {56 vars in 1109 bytes} [Thu Oct 20 08:48:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:45.297674 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d1592a4-b025-472d-a99c-46a7bfbae27b None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:45.330437 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3dae591-df9c-4867-97de-c27cd234befe None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:45.370801 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce67a010-cb98-434e-86fd-de711dab7d41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:45.371435 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce67a010-cb98-434e-86fd-de711dab7d41 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:45.371958 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce67a010-cb98-434e-86fd-de711dab7d41 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:45.372230 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce67a010-cb98-434e-86fd-de711dab7d41 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:45.386374 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce67a010-cb98-434e-86fd-de711dab7d41 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 20 08:48:45.392829 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-ce67a010-cb98-434e-86fd-de711dab7d41 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 20 08:48:45.392829 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:45.392829 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:45.392829 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:45.392829 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:45.392829 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:45.392829 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:45.392829 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:45.392829 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:45.392829 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:45.392829 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:45.392829 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:45.392829 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 20 08:48:45.392829 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 20 08:48:45.392829 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 20 08:48:45.392829 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 20 08:48:45.393768 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 20 08:48:45.393768 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 20 08:48:45.393768 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 20 08:48:45.393768 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 20 08:48:45.393768 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:45.393768 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:45.393768 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:45.393768 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:45.393768 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:45.393768 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:45.393768 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:45.393768 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:45.393768 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:45.393768 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:45.393768 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:45.393768 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:45.394650 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:45.394650 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:45.394650 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:45.394650 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:45.394650 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 20 08:48:45.394650 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 20 08:48:45.394650 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 20 08:48:45.394650 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 20 08:48:45.394650 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 20 08:48:45.394650 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 20 08:48:45.394650 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 20 08:48:45.394650 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:48:45.394650 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 271/544] 10.222.0.36 () {56 vars in 1102 bytes} [Thu Oct 20 08:48:45 2022] GET /identity/v3/roles/service => generated 85 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 20 08:48:45.396454 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce67a010-cb98-434e-86fd-de711dab7d41 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:45.409550 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:45.411345 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d1592a4-b025-472d-a99c-46a7bfbae27b None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:45.422612 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:45.448110 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70d5b90e-f80d-4621-8ff0-683e8a9d14c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:45.448643 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d5b90e-f80d-4621-8ff0-683e8a9d14c5 None admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:45.448921 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d5b90e-f80d-4621-8ff0-683e8a9d14c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:45.449189 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d5b90e-f80d-4621-8ff0-683e8a9d14c5 None admin] PATH_INFO: `/v3/services` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:45.449823 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70d5b90e-f80d-4621-8ff0-683e8a9d14c5 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:45.451219 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70d5b90e-f80d-4621-8ff0-683e8a9d14c5 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:45.467048 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 274/545] 10.222.0.36 () {60 vars in 1141 bytes} [Thu Oct 20 08:48:45 2022] POST /identity/v3/services => generated 251 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 08:48:45.468407 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261a15f7-52ee-46b7-b6ab-17f70940f55f None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:45.468695 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261a15f7-52ee-46b7-b6ab-17f70940f55f None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:45.468897 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261a15f7-52ee-46b7-b6ab-17f70940f55f None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:45.474587 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 275/546] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:48:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:48:45.474587 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-261a15f7-52ee-46b7-b6ab-17f70940f55f None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:45.506693 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce67a010-cb98-434e-86fd-de711dab7d41 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:45.551549 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a582adb-e339-4344-921d-61cce3269d0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:45.552876 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a582adb-e339-4344-921d-61cce3269d0a None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:45.553188 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a582adb-e339-4344-921d-61cce3269d0a None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:45.553465 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a582adb-e339-4344-921d-61cce3269d0a None admin] PATH_INFO: `/v3/projects/service` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:45.560886 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a582adb-e339-4344-921d-61cce3269d0a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:45.562385 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a582adb-e339-4344-921d-61cce3269d0a None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:45.574827 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-5a582adb-e339-4344-921d-61cce3269d0a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 20 08:48:45.574827 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:45.574827 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:45.574827 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:45.574827 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:45.574827 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:45.574827 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:45.574827 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:45.574827 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:45.574827 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:45.574827 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:45.574827 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:45.574827 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 20 08:48:45.574827 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 20 08:48:45.574827 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 20 08:48:45.575809 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 20 08:48:45.575809 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:45.575809 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:45.575809 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:45.575809 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:45.575809 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:45.575809 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:45.575809 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:45.575809 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:45.575809 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:45.575809 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:45.575809 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:45.575809 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:45.575809 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:45.575809 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:45.575809 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:45.575809 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:45.576692 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 20 08:48:45.576692 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 20 08:48:45.576692 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 20 08:48:45.576692 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 20 08:48:45.576692 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 20 08:48:45.576692 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 20 08:48:45.576692 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 20 08:48:45.576692 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:48:45.576692 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 272/547] 10.222.0.36 () {56 vars in 1111 bytes} [Thu Oct 20 08:48:45 2022] GET /identity/v3/projects/service => generated 88 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:45.577666 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8bd3353-3130-4d82-8c3c-bf0537d8efac None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:45.577923 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8bd3353-3130-4d82-8c3c-bf0537d8efac None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:45.578173 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8bd3353-3130-4d82-8c3c-bf0537d8efac None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:45.579636 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 273/548] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:48:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:48:45.580921 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f084ae53-6916-466a-98d8-f286ed98437a None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:45.581201 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f084ae53-6916-466a-98d8-f286ed98437a None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:45.581466 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f084ae53-6916-466a-98d8-f286ed98437a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:45.603881 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-f084ae53-6916-466a-98d8-f286ed98437a None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:48:45.605088 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-f084ae53-6916-466a-98d8-f286ed98437a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:48:45.606026 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-261a15f7-52ee-46b7-b6ab-17f70940f55f None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:45.643145 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb46d9d4-2cd0-42e6-938a-f608fb1bb1d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:45.643749 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb46d9d4-2cd0-42e6-938a-f608fb1bb1d5 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:45.644025 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb46d9d4-2cd0-42e6-938a-f608fb1bb1d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:45.644291 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb46d9d4-2cd0-42e6-938a-f608fb1bb1d5 None admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:45.644953 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb46d9d4-2cd0-42e6-938a-f608fb1bb1d5 None admin] RBAC: Adding query filter params (name=service) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:45.645248 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb46d9d4-2cd0-42e6-938a-f608fb1bb1d5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:45.646746 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb46d9d4-2cd0-42e6-938a-f608fb1bb1d5 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:45.656188 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 276/549] 10.222.0.36 () {56 vars in 1103 bytes} [Thu Oct 20 08:48:45 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:45.657186 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb46d9d4-2cd0-42e6-938a-f608fb1bb1d5 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:45.680793 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:45.690085 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 274/550] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:48:45 2022] POST /identity/v3/auth/tokens => generated 903 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 20 08:48:45.691191 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4929e7bb-269c-4bee-9f06-40cb3bac6ea9 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:45.691476 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4929e7bb-269c-4bee-9f06-40cb3bac6ea9 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:45.691747 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4929e7bb-269c-4bee-9f06-40cb3bac6ea9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:45.699423 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:45.711725 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-4929e7bb-269c-4bee-9f06-40cb3bac6ea9 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:48:45.712789 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-4929e7bb-269c-4bee-9f06-40cb3bac6ea9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:48:45.772405 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb46d9d4-2cd0-42e6-938a-f608fb1bb1d5 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:45.790670 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 275/551] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:48:45 2022] POST /identity/v3/auth/tokens => generated 903 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 20 08:48:45.790670 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4929e7bb-269c-4bee-9f06-40cb3bac6ea9 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:45.829572 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bff99b5-faca-4af5-a49f-b82c76492614 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:45.830576 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bff99b5-faca-4af5-a49f-b82c76492614 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:45.831062 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bff99b5-faca-4af5-a49f-b82c76492614 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:45.831557 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bff99b5-faca-4af5-a49f-b82c76492614 None admin] PATH_INFO: `/v3/projects` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:45.832449 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bff99b5-faca-4af5-a49f-b82c76492614 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:45.833073 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bff99b5-faca-4af5-a49f-b82c76492614 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:45.835520 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bff99b5-faca-4af5-a49f-b82c76492614 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:45.848163 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 277/552] 10.222.0.36 () {56 vars in 1148 bytes} [Thu Oct 20 08:48:45 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:45.859450 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bff99b5-faca-4af5-a49f-b82c76492614 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:45.949299 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4929e7bb-269c-4bee-9f06-40cb3bac6ea9 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:45.985028 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:45.997076 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5908c3db-76b1-41fd-a4c6-7d226c9c320c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:45.997699 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5908c3db-76b1-41fd-a4c6-7d226c9c320c None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:45.997972 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5908c3db-76b1-41fd-a4c6-7d226c9c320c None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:45.998236 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5908c3db-76b1-41fd-a4c6-7d226c9c320c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:46.012053 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5908c3db-76b1-41fd-a4c6-7d226c9c320c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:46.013519 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5908c3db-76b1-41fd-a4c6-7d226c9c320c None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:46.018321 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:46.029316 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bff99b5-faca-4af5-a49f-b82c76492614 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:46.032704 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-5908c3db-76b1-41fd-a4c6-7d226c9c320c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:46.032704 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:46.032704 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:46.032704 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:46.032704 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:46.032704 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:46.032704 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:46.032704 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:46.032704 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:46.032704 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:46.032704 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:46.032704 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:46.032704 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 20 08:48:46.032704 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 20 08:48:46.032704 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 20 08:48:46.032704 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 20 08:48:46.033674 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:46.033674 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:46.033674 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:46.033674 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:46.033674 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:46.033674 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:46.033674 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:46.033674 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:46.033674 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:46.033674 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:46.033674 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:46.033674 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:46.033674 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:46.033674 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:46.033674 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:46.033674 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:46.033674 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 20 08:48:46.034597 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 20 08:48:46.034597 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:46.034597 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:48:46.034597 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 276/553] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:48:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 247 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:46.034842 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5908c3db-76b1-41fd-a4c6-7d226c9c320c None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:46.071327 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c9bd9fe-6490-4afb-a996-a2a2ef45c5e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:46.071987 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c9bd9fe-6490-4afb-a996-a2a2ef45c5e4 None admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:46.072263 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c9bd9fe-6490-4afb-a996-a2a2ef45c5e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:46.072529 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c9bd9fe-6490-4afb-a996-a2a2ef45c5e4 None admin] PATH_INFO: `/v3/projects/d9857b610fb542969cfb5efbbdb8ce61/users/4d558918cd8b4ebea4fe8765297304db/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:46.075831 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c9bd9fe-6490-4afb-a996-a2a2ef45c5e4 None admin] RBAC: Authorizing `identity:create_grant(project_id=d9857b610fb542969cfb5efbbdb8ce61, user_id=4d558918cd8b4ebea4fe8765297304db, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:46.077484 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c9bd9fe-6490-4afb-a996-a2a2ef45c5e4 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:46.093806 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 278/554] 10.222.0.36 () {58 vars in 1439 bytes} [Thu Oct 20 08:48:45 2022] PUT /identity/v3/projects/d9857b610fb542969cfb5efbbdb8ce61/users/4d558918cd8b4ebea4fe8765297304db/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 234 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 08:48:46.103486 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c9bd9fe-6490-4afb-a996-a2a2ef45c5e4 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:46.160841 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5908c3db-76b1-41fd-a4c6-7d226c9c320c None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:46.212204 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abe4f78e-18ea-4b26-8865-50058b746296 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:46.212896 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe4f78e-18ea-4b26-8865-50058b746296 None admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:46.213178 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe4f78e-18ea-4b26-8865-50058b746296 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:46.213449 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe4f78e-18ea-4b26-8865-50058b746296 None admin] PATH_INFO: `/v3/services` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:46.214111 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abe4f78e-18ea-4b26-8865-50058b746296 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:46.216520 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abe4f78e-18ea-4b26-8865-50058b746296 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:46.241842 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 277/555] 10.222.0.36 () {60 vars in 1141 bytes} [Thu Oct 20 08:48:46 2022] POST /identity/v3/services => generated 240 bytes in 207 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 08:48:46.242914 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abe4f78e-18ea-4b26-8865-50058b746296 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:46.248847 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c9bd9fe-6490-4afb-a996-a2a2ef45c5e4 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:46.297871 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:46.347823 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4baeb34-7c95-4219-b289-a00db8498c69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:46.348431 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4baeb34-7c95-4219-b289-a00db8498c69 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:46.348707 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4baeb34-7c95-4219-b289-a00db8498c69 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:46.349036 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4baeb34-7c95-4219-b289-a00db8498c69 None admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:46.349677 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4baeb34-7c95-4219-b289-a00db8498c69 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:46.349976 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4baeb34-7c95-4219-b289-a00db8498c69 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:46.355533 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4baeb34-7c95-4219-b289-a00db8498c69 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:46.388844 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abe4f78e-18ea-4b26-8865-50058b746296 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:46.389364 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 279/556] 10.222.0.36 () {56 vars in 1109 bytes} [Thu Oct 20 08:48:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 286 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:46.399548 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4baeb34-7c95-4219-b289-a00db8498c69 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:46.429520 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0bfee41-5f83-4b9d-ab09-81596e6d4476 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:46.430183 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0bfee41-5f83-4b9d-ab09-81596e6d4476 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:46.430457 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0bfee41-5f83-4b9d-ab09-81596e6d4476 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:46.430750 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0bfee41-5f83-4b9d-ab09-81596e6d4476 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:46.437209 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:46.447660 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0bfee41-5f83-4b9d-ab09-81596e6d4476 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 20 08:48:46.452781 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-a0bfee41-5f83-4b9d-ab09-81596e6d4476 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 20 08:48:46.452781 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:46.452781 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:46.452781 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:46.452781 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:46.452781 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:46.452781 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:46.452781 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:46.452781 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:46.452781 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:46.452781 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:46.452781 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:46.452781 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 20 08:48:46.452781 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 20 08:48:46.452781 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 20 08:48:46.452781 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 20 08:48:46.453781 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 20 08:48:46.453781 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 20 08:48:46.453781 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 20 08:48:46.453781 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 20 08:48:46.453781 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:46.453781 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:46.453781 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:46.453781 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:46.453781 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:46.453781 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:46.453781 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:46.453781 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:46.453781 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:46.453781 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:46.453781 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:46.453781 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:46.456916 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:46.456916 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:46.456916 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:46.456916 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:46.456916 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 20 08:48:46.456916 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 20 08:48:46.456916 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 20 08:48:46.456916 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 20 08:48:46.456916 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 20 08:48:46.456916 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 20 08:48:46.456916 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 20 08:48:46.456916 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:48:46.456916 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 278/557] 10.222.0.36 () {56 vars in 1096 bytes} [Thu Oct 20 08:48:46 2022] GET /identity/v3/roles/admin => generated 83 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:46.465187 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0bfee41-5f83-4b9d-ab09-81596e6d4476 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:46.590936 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:46.596696 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0bfee41-5f83-4b9d-ab09-81596e6d4476 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:46.607069 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4baeb34-7c95-4219-b289-a00db8498c69 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:46.641164 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa3199fc-9fc9-4cc6-9477-2d541eee750f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:46.642031 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa3199fc-9fc9-4cc6-9477-2d541eee750f None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:46.642311 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa3199fc-9fc9-4cc6-9477-2d541eee750f None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:46.642612 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa3199fc-9fc9-4cc6-9477-2d541eee750f None admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:46.643274 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa3199fc-9fc9-4cc6-9477-2d541eee750f None admin] RBAC: Adding query filter params (name=admin) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:46.643571 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa3199fc-9fc9-4cc6-9477-2d541eee750f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:46.645181 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa3199fc-9fc9-4cc6-9477-2d541eee750f None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:46.652359 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-676af724-0cf2-4210-826f-9fb931429fd7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:46.653266 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676af724-0cf2-4210-826f-9fb931429fd7 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:46.653747 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676af724-0cf2-4210-826f-9fb931429fd7 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:46.654247 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676af724-0cf2-4210-826f-9fb931429fd7 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:46.658495 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 279/558] 10.222.0.36 () {56 vars in 1099 bytes} [Thu Oct 20 08:48:46 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 08:48:46.660764 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa3199fc-9fc9-4cc6-9477-2d541eee750f None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:46.667031 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-676af724-0cf2-4210-826f-9fb931429fd7 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:46.668584 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-676af724-0cf2-4210-826f-9fb931429fd7 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:46.680404 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-676af724-0cf2-4210-826f-9fb931429fd7 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 20 08:48:46.680404 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:46.680404 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:46.680404 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:46.680404 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:46.680404 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:46.680404 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:46.680404 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:46.680404 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:46.680404 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:46.680404 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:46.680404 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:46.680404 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 20 08:48:46.680404 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 20 08:48:46.680404 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 20 08:48:46.680404 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 20 08:48:46.681394 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:46.681394 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:46.681394 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 20 08:48:46.681394 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 20 08:48:46.681394 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 20 08:48:46.681394 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 20 08:48:46.681394 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:46.681394 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:46.681394 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:46.681394 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:46.681394 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:46.681394 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:46.681394 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:46.681394 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:46.681394 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:46.681394 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:46.681394 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:46.681394 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:46.682429 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:46.682429 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:46.682429 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 20 08:48:46.682429 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 20 08:48:46.682429 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 20 08:48:46.682429 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 20 08:48:46.682429 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 20 08:48:46.682429 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 20 08:48:46.682429 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 20 08:48:46.682429 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:46.682429 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 280/559] 10.222.0.36 () {56 vars in 1117 bytes} [Thu Oct 20 08:48:46 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 283 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:46.689093 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-676af724-0cf2-4210-826f-9fb931429fd7 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:46.793817 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa3199fc-9fc9-4cc6-9477-2d541eee750f None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:46.815481 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:46.857136 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03d9ba59-5d28-49b9-812e-b4772f47c9e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:46.857834 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d9ba59-5d28-49b9-812e-b4772f47c9e9 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:46.858114 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d9ba59-5d28-49b9-812e-b4772f47c9e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:46.858381 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d9ba59-5d28-49b9-812e-b4772f47c9e9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:46.872634 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-676af724-0cf2-4210-826f-9fb931429fd7 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:46.882328 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03d9ba59-5d28-49b9-812e-b4772f47c9e9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:46.883877 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03d9ba59-5d28-49b9-812e-b4772f47c9e9 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:46.905667 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-03d9ba59-5d28-49b9-812e-b4772f47c9e9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:46.905667 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:46.905667 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:46.905667 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:46.905667 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:46.905667 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:46.905667 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:46.905667 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:46.905667 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:46.905667 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:46.905667 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:46.905667 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:46.905667 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 20 08:48:46.905667 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 20 08:48:46.905667 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 20 08:48:46.905667 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 20 08:48:46.906701 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:46.906701 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:46.906701 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:46.906701 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:46.906701 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:46.906701 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:46.906701 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:46.906701 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:46.906701 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:46.906701 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:46.906701 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:46.906701 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:46.906701 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:46.906701 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:46.906701 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:46.906701 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:46.906701 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 20 08:48:46.907591 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 20 08:48:46.907591 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:46.907591 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:48:46.912724 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 280/560] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:48:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 251 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 20 08:48:46.917237 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03d9ba59-5d28-49b9-812e-b4772f47c9e9 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:46.941043 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d87f3e23-c3af-4e34-93a9-8deddd0aa6bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:46.941631 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d87f3e23-c3af-4e34-93a9-8deddd0aa6bc None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:46.941910 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d87f3e23-c3af-4e34-93a9-8deddd0aa6bc None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:46.942180 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d87f3e23-c3af-4e34-93a9-8deddd0aa6bc None admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:46.943517 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d87f3e23-c3af-4e34-93a9-8deddd0aa6bc None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:46.943818 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d87f3e23-c3af-4e34-93a9-8deddd0aa6bc None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:46.945243 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d87f3e23-c3af-4e34-93a9-8deddd0aa6bc None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:46.950364 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:46.977298 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 281/561] 10.222.0.36 () {56 vars in 1149 bytes} [Thu Oct 20 08:48:46 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 288 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:46.988208 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d87f3e23-c3af-4e34-93a9-8deddd0aa6bc None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:47.070387 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03d9ba59-5d28-49b9-812e-b4772f47c9e9 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:47.138115 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3802ada0-30ce-43c7-861a-3597e0a04335 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:47.140308 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3802ada0-30ce-43c7-861a-3597e0a04335 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:47.140992 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3802ada0-30ce-43c7-861a-3597e0a04335 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:47.142109 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3802ada0-30ce-43c7-861a-3597e0a04335 None admin] PATH_INFO: `/v3/domains` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:47.143159 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3802ada0-30ce-43c7-861a-3597e0a04335 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:47.146251 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3802ada0-30ce-43c7-861a-3597e0a04335 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:47.147876 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3802ada0-30ce-43c7-861a-3597e0a04335 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:47.170098 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 281/562] 10.222.0.36 () {56 vars in 1109 bytes} [Thu Oct 20 08:48:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 253 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:47.173205 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d87f3e23-c3af-4e34-93a9-8deddd0aa6bc None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:47.176042 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3802ada0-30ce-43c7-861a-3597e0a04335 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:47.234827 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:47.256109 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c94413ed-8e05-4629-9085-b0892e098d90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:47.256747 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94413ed-8e05-4629-9085-b0892e098d90 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:47.257010 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94413ed-8e05-4629-9085-b0892e098d90 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:47.257280 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94413ed-8e05-4629-9085-b0892e098d90 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:47.285100 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c94413ed-8e05-4629-9085-b0892e098d90 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:47.287851 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c94413ed-8e05-4629-9085-b0892e098d90 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:47.288411 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:47.314058 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-c94413ed-8e05-4629-9085-b0892e098d90 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:47.314058 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:47.314058 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:47.314058 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:47.314058 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:47.314058 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:47.314058 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:47.314058 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:47.314058 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:47.314058 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:47.314058 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:47.314058 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:47.314058 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 20 08:48:47.314058 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 20 08:48:47.314058 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 20 08:48:47.314058 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 20 08:48:47.315243 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:47.315243 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:47.315243 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:47.315243 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:47.315243 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:47.315243 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:47.315243 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:47.315243 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:47.315243 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:47.315243 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:47.315243 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:47.315243 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:47.315243 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:47.315243 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:47.315243 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:47.315243 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:47.315243 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 20 08:48:47.316406 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 20 08:48:47.316406 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:47.316406 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:47.316406 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 282/563] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:48:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 332 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:47.325491 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c94413ed-8e05-4629-9085-b0892e098d90 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:47.346119 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3802ada0-30ce-43c7-861a-3597e0a04335 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:47.438374 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fde4624b-fe37-406e-ba55-b0f98dabdfd9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:47.439023 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fde4624b-fe37-406e-ba55-b0f98dabdfd9 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:47.439283 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fde4624b-fe37-406e-ba55-b0f98dabdfd9 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:47.439552 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fde4624b-fe37-406e-ba55-b0f98dabdfd9 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:47.463240 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fde4624b-fe37-406e-ba55-b0f98dabdfd9 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:47.464703 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fde4624b-fe37-406e-ba55-b0f98dabdfd9 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:47.465632 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c94413ed-8e05-4629-9085-b0892e098d90 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:47.487559 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-fde4624b-fe37-406e-ba55-b0f98dabdfd9 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 20 08:48:47.487559 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:47.487559 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:47.487559 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:47.487559 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:47.487559 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:47.487559 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:47.487559 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:47.487559 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:47.487559 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:47.487559 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:47.487559 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:47.487559 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 20 08:48:47.487559 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 20 08:48:47.487559 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 20 08:48:47.487559 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 20 08:48:47.488762 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:47.488762 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:47.488762 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 20 08:48:47.488762 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 20 08:48:47.488762 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 20 08:48:47.488762 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 20 08:48:47.488762 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:47.488762 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:47.488762 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:47.488762 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:47.488762 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:47.488762 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:47.488762 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:47.488762 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:47.488762 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:47.488762 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:47.488762 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:47.488762 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:47.490106 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:47.490106 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:47.490106 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 20 08:48:47.490106 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 20 08:48:47.490106 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 20 08:48:47.490106 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 20 08:48:47.490106 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 20 08:48:47.490106 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 20 08:48:47.490106 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 20 08:48:47.490106 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:48:47.490106 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 282/564] 10.222.0.36 () {56 vars in 1093 bytes} [Thu Oct 20 08:48:47 2022] GET /identity/v3/users/nova => generated 82 bytes in 313 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:47.494437 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fde4624b-fe37-406e-ba55-b0f98dabdfd9 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:47.505292 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c94f5fa5-74d4-468b-8a47-746de93e5f33 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:47.506116 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94f5fa5-74d4-468b-8a47-746de93e5f33 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:47.506559 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94f5fa5-74d4-468b-8a47-746de93e5f33 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:47.506998 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94f5fa5-74d4-468b-8a47-746de93e5f33 None admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:47.507776 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c94f5fa5-74d4-468b-8a47-746de93e5f33 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:47.508219 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c94f5fa5-74d4-468b-8a47-746de93e5f33 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:47.509738 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c94f5fa5-74d4-468b-8a47-746de93e5f33 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:47.523188 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 283/565] 10.222.0.36 () {56 vars in 1109 bytes} [Thu Oct 20 08:48:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:47.531380 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c94f5fa5-74d4-468b-8a47-746de93e5f33 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:47.608642 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:47.696579 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fde4624b-fe37-406e-ba55-b0f98dabdfd9 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:47.720647 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c94f5fa5-74d4-468b-8a47-746de93e5f33 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:47.758520 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:47.764660 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5617bae9-b945-427a-a0b5-b84609e589a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:47.765271 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5617bae9-b945-427a-a0b5-b84609e589a2 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:47.765636 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5617bae9-b945-427a-a0b5-b84609e589a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:47.766590 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5617bae9-b945-427a-a0b5-b84609e589a2 None admin] PATH_INFO: `/v3/users` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:47.767537 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5617bae9-b945-427a-a0b5-b84609e589a2 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:47.767842 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5617bae9-b945-427a-a0b5-b84609e589a2 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:47.769280 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5617bae9-b945-427a-a0b5-b84609e589a2 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:47.789533 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c017405b-0dae-4461-9a20-5acb9061be23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:47.790217 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c017405b-0dae-4461-9a20-5acb9061be23 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:47.790503 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c017405b-0dae-4461-9a20-5acb9061be23 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:47.790776 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c017405b-0dae-4461-9a20-5acb9061be23 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:47.801778 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 283/566] 10.222.0.36 () {56 vars in 1133 bytes} [Thu Oct 20 08:48:47 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 307 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:47.802914 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c017405b-0dae-4461-9a20-5acb9061be23 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:47.803175 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b98ce4-674e-4740-bca0-eaeaf599c5a3 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:47.803463 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b98ce4-674e-4740-bca0-eaeaf599c5a3 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:47.803729 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b98ce4-674e-4740-bca0-eaeaf599c5a3 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:47.804634 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c017405b-0dae-4461-9a20-5acb9061be23 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:47.805175 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 284/567] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:48:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:48:47.806719 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b98ce4-674e-4740-bca0-eaeaf599c5a3 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:47.818798 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-c017405b-0dae-4461-9a20-5acb9061be23 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 20 08:48:47.818798 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:47.818798 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:47.818798 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:47.818798 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:47.818798 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:47.818798 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:47.818798 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:47.818798 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:47.818798 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:47.818798 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:47.818798 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:47.818798 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 20 08:48:47.818798 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 20 08:48:47.818798 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 20 08:48:47.819777 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 20 08:48:47.819777 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:47.819777 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:47.819777 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:47.819777 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:47.819777 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:47.819777 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:47.819777 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:47.819777 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:47.819777 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:47.819777 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:47.819777 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:47.819777 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:47.819777 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:47.819777 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:47.819777 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:47.819777 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:47.820965 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 20 08:48:47.820965 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 20 08:48:47.820965 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 20 08:48:47.820965 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 20 08:48:47.820965 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 20 08:48:47.820965 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 20 08:48:47.820965 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 20 08:48:47.820965 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:47.824805 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 284/568] 10.222.0.36 () {56 vars in 1111 bytes} [Thu Oct 20 08:48:47 2022] GET /identity/v3/projects/service => generated 88 bytes in 289 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:47.826206 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42823011-c714-4c24-80b8-700ab5bf74e8 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:47.826589 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42823011-c714-4c24-80b8-700ab5bf74e8 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:47.826957 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42823011-c714-4c24-80b8-700ab5bf74e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:47.846228 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-42823011-c714-4c24-80b8-700ab5bf74e8 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:48:47.847648 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-42823011-c714-4c24-80b8-700ab5bf74e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:48:47.916827 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b98ce4-674e-4740-bca0-eaeaf599c5a3 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:47.926504 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 285/569] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:48:47 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 20 08:48:47.926504 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42823011-c714-4c24-80b8-700ab5bf74e8 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:47.977451 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:47.990749 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbc12cc6-3640-49f9-b6da-01a73586e66c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:47.991362 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc12cc6-3640-49f9-b6da-01a73586e66c None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:47.991634 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc12cc6-3640-49f9-b6da-01a73586e66c None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:47.991898 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc12cc6-3640-49f9-b6da-01a73586e66c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:48.109287 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:48.122630 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42823011-c714-4c24-80b8-700ab5bf74e8 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:48.128673 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbc12cc6-3640-49f9-b6da-01a73586e66c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:48.132027 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbc12cc6-3640-49f9-b6da-01a73586e66c None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:48.191518 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f5e3726-260d-47b9-a832-023d98695a5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:48.192639 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f5e3726-260d-47b9-a832-023d98695a5e None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:48.193152 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f5e3726-260d-47b9-a832-023d98695a5e None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:48.193733 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f5e3726-260d-47b9-a832-023d98695a5e None admin] PATH_INFO: `/v3/projects` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:48.194663 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f5e3726-260d-47b9-a832-023d98695a5e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:48.195215 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f5e3726-260d-47b9-a832-023d98695a5e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:48.197047 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f5e3726-260d-47b9-a832-023d98695a5e None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:48.226944 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 286/570] 10.222.0.36 () {56 vars in 1148 bytes} [Thu Oct 20 08:48:47 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 305 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:48.233332 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f5e3726-260d-47b9-a832-023d98695a5e None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:48.256413 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 285/571] 10.222.0.36 () {58 vars in 1282 bytes} [Thu Oct 20 08:48:47 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 450 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 20 08:48:48.267366 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbc12cc6-3640-49f9-b6da-01a73586e66c None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:48.352966 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:48.369000 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f5e3726-260d-47b9-a832-023d98695a5e None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:48.386204 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:48.398687 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbc12cc6-3640-49f9-b6da-01a73586e66c None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:48.412593 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06c9bc14-5d0f-4c9b-9cf5-a3fc1a2b46ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:48.413275 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c9bc14-5d0f-4c9b-9cf5-a3fc1a2b46ef None admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:48.413551 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c9bc14-5d0f-4c9b-9cf5-a3fc1a2b46ef None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:48.413820 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c9bc14-5d0f-4c9b-9cf5-a3fc1a2b46ef None admin] PATH_INFO: `/v3/projects/d9857b610fb542969cfb5efbbdb8ce61/users/f4b0e9187ce74f7888e9e7094be5c824/roles/f81100d13381464aa48cbae3561ca1bc` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:48.434489 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06c9bc14-5d0f-4c9b-9cf5-a3fc1a2b46ef None admin] RBAC: Authorizing `identity:create_grant(project_id=d9857b610fb542969cfb5efbbdb8ce61, user_id=f4b0e9187ce74f7888e9e7094be5c824, role_id=f81100d13381464aa48cbae3561ca1bc)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:48.436057 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06c9bc14-5d0f-4c9b-9cf5-a3fc1a2b46ef None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:48.437156 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9ef232-7b7b-441c-b4d0-566140aaaf0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:48.438015 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9ef232-7b7b-441c-b4d0-566140aaaf0b None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:48.438657 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9ef232-7b7b-441c-b4d0-566140aaaf0b None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:48.439608 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9ef232-7b7b-441c-b4d0-566140aaaf0b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:48.453382 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 287/572] 10.222.0.36 () {58 vars in 1439 bytes} [Thu Oct 20 08:48:48 2022] PUT /identity/v3/projects/d9857b610fb542969cfb5efbbdb8ce61/users/f4b0e9187ce74f7888e9e7094be5c824/roles/f81100d13381464aa48cbae3561ca1bc => generated 0 bytes in 220 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 08:48:48.453887 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad9ef232-7b7b-441c-b4d0-566140aaaf0b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:48.454253 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06c9bc14-5d0f-4c9b-9cf5-a3fc1a2b46ef None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:48.455901 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad9ef232-7b7b-441c-b4d0-566140aaaf0b None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:48.475081 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-ad9ef232-7b7b-441c-b4d0-566140aaaf0b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:48.475081 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:48.475081 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:48.475081 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:48.475081 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:48.475081 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:48.475081 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:48.475081 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:48.475081 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:48.475081 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:48.475081 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:48.475081 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:48.475081 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 20 08:48:48.475081 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 20 08:48:48.475081 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 20 08:48:48.475081 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 20 08:48:48.476167 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:48.476167 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:48.476167 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:48.476167 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:48.476167 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:48.476167 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:48.476167 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:48.476167 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:48.476167 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:48.476167 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:48.476167 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:48.476167 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:48.476167 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:48.476167 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:48.476167 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:48.476167 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:48.476167 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 20 08:48:48.477059 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 20 08:48:48.477059 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:48.477059 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:48:48.479392 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 286/573] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:48:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:48.485964 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9ef232-7b7b-441c-b4d0-566140aaaf0b None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:48.617804 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06c9bc14-5d0f-4c9b-9cf5-a3fc1a2b46ef None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:48.667743 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9ef232-7b7b-441c-b4d0-566140aaaf0b None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:48.690483 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58ab586b-75f6-4abd-aff5-79cad61813f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:48.691296 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ab586b-75f6-4abd-aff5-79cad61813f9 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:48.691577 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ab586b-75f6-4abd-aff5-79cad61813f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:48.691843 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ab586b-75f6-4abd-aff5-79cad61813f9 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:48.692489 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58ab586b-75f6-4abd-aff5-79cad61813f9 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:48.694290 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58ab586b-75f6-4abd-aff5-79cad61813f9 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:48.706322 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-58ab586b-75f6-4abd-aff5-79cad61813f9 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 20 08:48:48.706322 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:48.706322 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 20 08:48:48.706322 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 20 08:48:48.706322 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 20 08:48:48.706322 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 20 08:48:48.706322 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 20 08:48:48.706322 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 20 08:48:48.706322 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 20 08:48:48.706322 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:48.706322 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 20 08:48:48.706322 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:48.706322 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:48.706322 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:48.706322 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:48.706322 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:48.706322 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:48.707415 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:48.707415 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:48.707415 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:48.707415 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:48.707415 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:48.707415 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:48.707415 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 20 08:48:48.707415 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 20 08:48:48.707415 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 20 08:48:48.707415 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 20 08:48:48.707415 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:48.707415 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:48.707415 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:48.707415 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:48.707415 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:48.707415 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:48.708299 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:48.708299 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:48.708299 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:48.708299 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:48.708299 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:48.708299 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:48.708299 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:48.708299 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:48.708299 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:48.708299 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:48.708299 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 20 08:48:48.708299 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 20 08:48:48.708299 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 20 08:48:48.708299 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:48.709335 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 288/574] 10.222.0.36 () {56 vars in 1129 bytes} [Thu Oct 20 08:48:48 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 254 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:48.710092 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a97a36-6e95-4251-b0ee-d776dd24216f None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:48.710092 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a97a36-6e95-4251-b0ee-d776dd24216f None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:48.710338 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a97a36-6e95-4251-b0ee-d776dd24216f None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:48.711643 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 289/575] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:48:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:48:48.714232 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13a97a36-6e95-4251-b0ee-d776dd24216f None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:48.718383 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:48.719668 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a22f61b2-f3c1-4947-a2ba-182eeb036319 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:48.721469 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a22f61b2-f3c1-4947-a2ba-182eeb036319 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:48.721751 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a22f61b2-f3c1-4947-a2ba-182eeb036319 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:48.722022 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a22f61b2-f3c1-4947-a2ba-182eeb036319 None admin] PATH_INFO: `/v3/domains` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:48.722734 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a22f61b2-f3c1-4947-a2ba-182eeb036319 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:48.723035 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a22f61b2-f3c1-4947-a2ba-182eeb036319 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:48.724998 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a22f61b2-f3c1-4947-a2ba-182eeb036319 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:48.741519 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 287/576] 10.222.0.36 () {56 vars in 1109 bytes} [Thu Oct 20 08:48:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 256 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:48.746582 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbee3840-91a1-479b-bf48-c2659b625980 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:48.746897 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbee3840-91a1-479b-bf48-c2659b625980 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:48.747178 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbee3840-91a1-479b-bf48-c2659b625980 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:48.754055 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:48.771177 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-fbee3840-91a1-479b-bf48-c2659b625980 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:48:48.772384 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-fbee3840-91a1-479b-bf48-c2659b625980 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:48:48.846982 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13a97a36-6e95-4251-b0ee-d776dd24216f None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:48.866622 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 288/577] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:48:48 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 20 08:48:48.867870 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbee3840-91a1-479b-bf48-c2659b625980 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:48.900955 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d15ed256-53e4-41f0-9429-969104e4a629 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:48.901948 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d15ed256-53e4-41f0-9429-969104e4a629 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:48.902605 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d15ed256-53e4-41f0-9429-969104e4a629 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:48.903086 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d15ed256-53e4-41f0-9429-969104e4a629 None admin] PATH_INFO: `/v3/services` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:48.904970 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d15ed256-53e4-41f0-9429-969104e4a629 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:48.905656 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d15ed256-53e4-41f0-9429-969104e4a629 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:48.908400 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d15ed256-53e4-41f0-9429-969104e4a629 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:48.916758 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 290/578] 10.222.0.36 () {56 vars in 1124 bytes} [Thu Oct 20 08:48:48 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:48.918072 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7178411e-18d7-423b-945e-aa8f780aeda9 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:48.918351 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7178411e-18d7-423b-945e-aa8f780aeda9 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:48.918684 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7178411e-18d7-423b-945e-aa8f780aeda9 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:48.923948 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 291/579] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:48:48 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:48:48.924753 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7178411e-18d7-423b-945e-aa8f780aeda9 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:48.996146 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:49.014097 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbee3840-91a1-479b-bf48-c2659b625980 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:49.034766 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7178411e-18d7-423b-945e-aa8f780aeda9 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:49.057852 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:49.066438 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d86bcfd-423a-4ab7-9e1b-b565f6684e25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:49.067384 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d86bcfd-423a-4ab7-9e1b-b565f6684e25 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:49.067927 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d86bcfd-423a-4ab7-9e1b-b565f6684e25 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:49.068458 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d86bcfd-423a-4ab7-9e1b-b565f6684e25 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:49.074993 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea94486a-f4fe-4542-8322-f00233396886 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:49.075739 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea94486a-f4fe-4542-8322-f00233396886 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:49.076154 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea94486a-f4fe-4542-8322-f00233396886 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:49.076394 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea94486a-f4fe-4542-8322-f00233396886 None admin] PATH_INFO: `/v3/services` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:49.077181 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea94486a-f4fe-4542-8322-f00233396886 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:49.077546 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea94486a-f4fe-4542-8322-f00233396886 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:49.078049 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d86bcfd-423a-4ab7-9e1b-b565f6684e25 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:49.079123 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea94486a-f4fe-4542-8322-f00233396886 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:49.079934 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d86bcfd-423a-4ab7-9e1b-b565f6684e25 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:49.085309 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 292/580] 10.222.0.36 () {56 vars in 1124 bytes} [Thu Oct 20 08:48:48 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:49.086800 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05438b78-3752-4402-b55f-ce2666533769 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:49.087163 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05438b78-3752-4402-b55f-ce2666533769 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:49.087504 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05438b78-3752-4402-b55f-ce2666533769 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:49.094156 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-0d86bcfd-423a-4ab7-9e1b-b565f6684e25 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 20 08:48:49.094156 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:49.094156 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:49.094156 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:49.094156 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:49.094156 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:49.094156 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:49.094156 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:49.094156 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:49.094156 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:49.094156 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:49.094156 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:49.094156 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 20 08:48:49.094156 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 20 08:48:49.094156 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 20 08:48:49.095247 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 20 08:48:49.095247 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:49.095247 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:49.095247 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:49.095247 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:49.095247 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:49.095247 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:49.095247 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:49.095247 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:49.095247 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:49.095247 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:49.095247 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:49.095247 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:49.095247 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:49.095247 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:49.095247 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:49.095247 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:49.096412 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 20 08:48:49.096412 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 20 08:48:49.096412 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 20 08:48:49.096412 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 20 08:48:49.096412 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 20 08:48:49.096412 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 20 08:48:49.096412 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 20 08:48:49.096412 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:48:49.099471 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 289/581] 10.222.0.36 () {56 vars in 1111 bytes} [Thu Oct 20 08:48:48 2022] GET /identity/v3/projects/service => generated 88 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:49.100532 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d86bcfd-423a-4ab7-9e1b-b565f6684e25 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:49.107797 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-05438b78-3752-4402-b55f-ce2666533769 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:48:49.109033 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-05438b78-3752-4402-b55f-ce2666533769 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:48:49.178154 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 293/582] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:48:49 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 20 08:48:49.184964 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05438b78-3752-4402-b55f-ce2666533769 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:49.215878 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d86bcfd-423a-4ab7-9e1b-b565f6684e25 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:49.252582 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:49.263745 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b32918bd-aeb9-420a-9b32-aa628f2fd877 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:49.264362 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b32918bd-aeb9-420a-9b32-aa628f2fd877 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:49.264640 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b32918bd-aeb9-420a-9b32-aa628f2fd877 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:49.264919 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b32918bd-aeb9-420a-9b32-aa628f2fd877 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:49.265886 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b32918bd-aeb9-420a-9b32-aa628f2fd877 None admin] RBAC: Adding query filter params (interface=public, service_id=2064157e30b446e0b4a1497a7ebf9f2b, region_id=RegionOne) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:49.266195 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b32918bd-aeb9-420a-9b32-aa628f2fd877 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:49.268755 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b32918bd-aeb9-420a-9b32-aa628f2fd877 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:49.281885 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 290/583] 10.222.0.36 () {56 vars in 1251 bytes} [Thu Oct 20 08:48:49 2022] GET /identity/v3/endpoints?service_id=2064157e30b446e0b4a1497a7ebf9f2b&interface=public®ion_id=RegionOne => generated 194 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:49.283103 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b32918bd-aeb9-420a-9b32-aa628f2fd877 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:49.302139 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05438b78-3752-4402-b55f-ce2666533769 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:49.349435 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdb39180-080c-44ed-a315-dda7a1b4aa19 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:49.350055 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb39180-080c-44ed-a315-dda7a1b4aa19 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:49.350332 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb39180-080c-44ed-a315-dda7a1b4aa19 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:49.350635 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb39180-080c-44ed-a315-dda7a1b4aa19 None admin] PATH_INFO: `/v3/projects` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:49.351315 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdb39180-080c-44ed-a315-dda7a1b4aa19 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:49.351611 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdb39180-080c-44ed-a315-dda7a1b4aa19 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:49.353084 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdb39180-080c-44ed-a315-dda7a1b4aa19 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:49.358830 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:49.366510 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 294/584] 10.222.0.36 () {56 vars in 1148 bytes} [Thu Oct 20 08:48:49 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:49.367340 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdb39180-080c-44ed-a315-dda7a1b4aa19 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:49.412714 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b32918bd-aeb9-420a-9b32-aa628f2fd877 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:49.456680 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff5a79d7-4e08-4528-a5f1-5748e4c7d385 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:49.457320 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff5a79d7-4e08-4528-a5f1-5748e4c7d385 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:49.457560 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff5a79d7-4e08-4528-a5f1-5748e4c7d385 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:49.457818 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff5a79d7-4e08-4528-a5f1-5748e4c7d385 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:49.471880 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff5a79d7-4e08-4528-a5f1-5748e4c7d385 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 20 08:48:49.479724 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-ff5a79d7-4e08-4528-a5f1-5748e4c7d385 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 20 08:48:49.479724 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:49.479724 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:49.479724 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:49.479724 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:49.479724 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:49.479724 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:49.479724 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:49.479724 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:49.479724 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:49.479724 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:49.479724 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:49.479724 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 20 08:48:49.479724 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 20 08:48:49.479724 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 20 08:48:49.479724 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 20 08:48:49.480736 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 20 08:48:49.480736 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 20 08:48:49.480736 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 20 08:48:49.480736 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 20 08:48:49.480736 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:49.480736 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:49.480736 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:49.480736 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:49.480736 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:49.480736 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:49.480736 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:49.480736 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:49.480736 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:49.480736 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:49.480736 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:49.480736 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:49.481660 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:49.481660 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:49.481660 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:49.481660 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:49.481660 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 20 08:48:49.481660 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 20 08:48:49.481660 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 20 08:48:49.481660 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 20 08:48:49.481660 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 20 08:48:49.481660 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 20 08:48:49.481660 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 20 08:48:49.481660 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:48:49.481660 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 291/585] 10.222.0.36 () {56 vars in 1096 bytes} [Thu Oct 20 08:48:49 2022] GET /identity/v3/roles/admin => generated 83 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:49.482356 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff5a79d7-4e08-4528-a5f1-5748e4c7d385 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:49.482433 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdb39180-080c-44ed-a315-dda7a1b4aa19 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:49.504569 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:49.520379 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30a3846c-8952-432c-ad8f-02762ce6d0f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:49.521025 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a3846c-8952-432c-ad8f-02762ce6d0f5 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:49.521308 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a3846c-8952-432c-ad8f-02762ce6d0f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:49.521583 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a3846c-8952-432c-ad8f-02762ce6d0f5 None admin] PATH_INFO: `/v3/services` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:49.522190 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30a3846c-8952-432c-ad8f-02762ce6d0f5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:49.523615 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30a3846c-8952-432c-ad8f-02762ce6d0f5 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:49.528755 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 295/586] 10.222.0.36 () {56 vars in 1087 bytes} [Thu Oct 20 08:48:49 2022] GET /identity/v3/services => generated 772 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:49.529531 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30a3846c-8952-432c-ad8f-02762ce6d0f5 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:49.607961 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff5a79d7-4e08-4528-a5f1-5748e4c7d385 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:49.632169 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:49.648887 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c478eed4-c3b5-4aed-bbd4-ccb39b42e33f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:49.650021 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c478eed4-c3b5-4aed-bbd4-ccb39b42e33f None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:49.650453 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c478eed4-c3b5-4aed-bbd4-ccb39b42e33f None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:49.650903 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c478eed4-c3b5-4aed-bbd4-ccb39b42e33f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:49.651820 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c478eed4-c3b5-4aed-bbd4-ccb39b42e33f None admin] RBAC: Adding query filter params (role.id=d9d3bc62f6e8453488c74856ef333bb3, scope.project.id=d9857b610fb542969cfb5efbbdb8ce61, user.id=a94e868115ec49918eda10724263930c) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:49.652288 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c478eed4-c3b5-4aed-bbd4-ccb39b42e33f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:49.654124 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c478eed4-c3b5-4aed-bbd4-ccb39b42e33f None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:49.660573 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30a3846c-8952-432c-ad8f-02762ce6d0f5 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:49.662272 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 292/587] 10.222.0.36 () {56 vars in 1374 bytes} [Thu Oct 20 08:48:49 2022] GET /identity/v3/role_assignments?user.id=a94e868115ec49918eda10724263930c&scope.project.id=d9857b610fb542969cfb5efbbdb8ce61&role.id=d9d3bc62f6e8453488c74856ef333bb3 => generated 608 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:49.663466 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c478eed4-c3b5-4aed-bbd4-ccb39b42e33f None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:49.741821 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94326442-3119-4d44-993f-222de017ae98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:49.742778 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94326442-3119-4d44-993f-222de017ae98 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:49.743210 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94326442-3119-4d44-993f-222de017ae98 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:49.743661 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94326442-3119-4d44-993f-222de017ae98 None admin] PATH_INFO: `/v3/services/network` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:49.744431 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94326442-3119-4d44-993f-222de017ae98 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:49.746032 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94326442-3119-4d44-993f-222de017ae98 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:49.762599 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-94326442-3119-4d44-993f-222de017ae98 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Oct 20 08:48:49.762599 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:49.762599 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 20 08:48:49.762599 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 20 08:48:49.762599 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 20 08:48:49.762599 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 20 08:48:49.762599 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 20 08:48:49.762599 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 20 08:48:49.762599 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 20 08:48:49.762599 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:49.762599 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 20 08:48:49.762599 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:49.762599 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:49.762599 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:49.762599 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:49.762599 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:49.762599 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:49.763913 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:49.763913 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:49.763913 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:49.763913 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:49.763913 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:49.763913 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:49.763913 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 20 08:48:49.763913 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 20 08:48:49.763913 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 20 08:48:49.763913 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 20 08:48:49.763913 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:49.763913 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:49.763913 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:49.763913 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:49.763913 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:49.763913 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:49.764960 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:49.764960 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:49.764960 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:49.764960 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:49.764960 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:49.764960 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:49.764960 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:49.764960 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:49.764960 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:49.764960 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:49.764960 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 20 08:48:49.764960 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 20 08:48:49.764960 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 20 08:48:49.764960 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:49.768983 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 296/588] 10.222.0.36 () {56 vars in 1111 bytes} [Thu Oct 20 08:48:49 2022] GET /identity/v3/services/network => generated 88 bytes in 240 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:49.778176 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94326442-3119-4d44-993f-222de017ae98 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:49.851717 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:49.861847 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c478eed4-c3b5-4aed-bbd4-ccb39b42e33f None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:49.910198 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12d254d9-a81b-494b-9824-c978b2c428af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:49.910945 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d254d9-a81b-494b-9824-c978b2c428af None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:49.911329 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d254d9-a81b-494b-9824-c978b2c428af None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:49.911708 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d254d9-a81b-494b-9824-c978b2c428af None admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:49.912701 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12d254d9-a81b-494b-9824-c978b2c428af None admin] RBAC: Adding query filter params (name=admin) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:49.913070 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12d254d9-a81b-494b-9824-c978b2c428af None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:49.915261 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12d254d9-a81b-494b-9824-c978b2c428af None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:49.916556 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94326442-3119-4d44-993f-222de017ae98 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:49.930543 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 293/589] 10.222.0.36 () {56 vars in 1099 bytes} [Thu Oct 20 08:48:49 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 263 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:49.933048 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12d254d9-a81b-494b-9824-c978b2c428af None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:49.965383 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a637ca7-ad0f-4157-a87d-dc15fcaed247 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:49.965988 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a637ca7-ad0f-4157-a87d-dc15fcaed247 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:49.966262 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a637ca7-ad0f-4157-a87d-dc15fcaed247 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:49.966554 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a637ca7-ad0f-4157-a87d-dc15fcaed247 None admin] PATH_INFO: `/v3/services` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:49.967425 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a637ca7-ad0f-4157-a87d-dc15fcaed247 None admin] RBAC: Adding query filter params (name=network) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:49.967730 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a637ca7-ad0f-4157-a87d-dc15fcaed247 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:49.969806 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a637ca7-ad0f-4157-a87d-dc15fcaed247 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:49.981002 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 297/590] 10.222.0.36 () {56 vars in 1112 bytes} [Thu Oct 20 08:48:49 2022] GET /identity/v3/services?name=network => generated 124 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:49.983809 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a637ca7-ad0f-4157-a87d-dc15fcaed247 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:49.998503 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:50.125981 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12d254d9-a81b-494b-9824-c978b2c428af None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:50.141697 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a637ca7-ad0f-4157-a87d-dc15fcaed247 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:50.167353 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:50.186288 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4749151-4ca2-47f0-ba54-f7bc03f1f99f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:50.186940 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4749151-4ca2-47f0-ba54-f7bc03f1f99f None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:50.187208 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4749151-4ca2-47f0-ba54-f7bc03f1f99f None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:50.187472 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4749151-4ca2-47f0-ba54-f7bc03f1f99f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:50.205687 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4749151-4ca2-47f0-ba54-f7bc03f1f99f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:50.207185 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4749151-4ca2-47f0-ba54-f7bc03f1f99f None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:50.211730 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0088172-1b8b-4434-8059-734f059b27f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:50.212330 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0088172-1b8b-4434-8059-734f059b27f4 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:50.212602 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0088172-1b8b-4434-8059-734f059b27f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:50.212869 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0088172-1b8b-4434-8059-734f059b27f4 None admin] PATH_INFO: `/v3/services` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:50.213494 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0088172-1b8b-4434-8059-734f059b27f4 None admin] RBAC: Adding query filter params (type=network) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:50.213780 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0088172-1b8b-4434-8059-734f059b27f4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:50.216261 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0088172-1b8b-4434-8059-734f059b27f4 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:50.227693 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 298/591] 10.222.0.36 () {56 vars in 1112 bytes} [Thu Oct 20 08:48:49 2022] GET /identity/v3/services?type=network => generated 350 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 08:48:50.239080 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-a4749151-4ca2-47f0-ba54-f7bc03f1f99f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:50.239080 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:50.239080 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:50.239080 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:50.239080 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:50.239080 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:50.239080 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:50.239080 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:50.239080 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:50.239080 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:50.239080 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:50.239080 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:50.239080 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 20 08:48:50.239080 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 20 08:48:50.239080 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 20 08:48:50.239080 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 20 08:48:50.240607 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:50.240607 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:50.240607 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:50.240607 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:50.240607 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:50.240607 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:50.240607 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:50.240607 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:50.240607 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:50.240607 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:50.240607 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:50.240607 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:50.240607 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:50.240607 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:50.240607 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:50.240607 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:50.240607 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 20 08:48:50.241518 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 20 08:48:50.241518 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:50.241518 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:48:50.241518 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 294/592] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:48:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 308 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:50.245490 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0088172-1b8b-4434-8059-734f059b27f4 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:50.251494 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4749151-4ca2-47f0-ba54-f7bc03f1f99f None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:50.355891 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0088172-1b8b-4434-8059-734f059b27f4 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:50.391157 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:50.398096 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e930a6b9-3c5b-4188-8631-d21cdbbd50cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:50.398096 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e930a6b9-3c5b-4188-8631-d21cdbbd50cb None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:50.398096 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e930a6b9-3c5b-4188-8631-d21cdbbd50cb None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:50.398096 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e930a6b9-3c5b-4188-8631-d21cdbbd50cb None admin] PATH_INFO: `/v3/endpoints` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:50.398813 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e930a6b9-3c5b-4188-8631-d21cdbbd50cb None admin] RBAC: Adding query filter params (interface=public, service_id=0f13dfd626b24199a547e8dd7b93ab9a, region_id=RegionOne) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:50.399122 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e930a6b9-3c5b-4188-8631-d21cdbbd50cb None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:50.401352 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e930a6b9-3c5b-4188-8631-d21cdbbd50cb None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:50.404615 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4749151-4ca2-47f0-ba54-f7bc03f1f99f None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:50.424715 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 299/593] 10.222.0.36 () {56 vars in 1251 bytes} [Thu Oct 20 08:48:50 2022] GET /identity/v3/endpoints?service_id=0f13dfd626b24199a547e8dd7b93ab9a&interface=public®ion_id=RegionOne => generated 194 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 08:48:50.435456 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e930a6b9-3c5b-4188-8631-d21cdbbd50cb None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:50.448329 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e5b2946-4ac2-4ddf-b681-ef10f00c163c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:50.448974 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5b2946-4ac2-4ddf-b681-ef10f00c163c None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:50.449246 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5b2946-4ac2-4ddf-b681-ef10f00c163c None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:50.449563 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5b2946-4ac2-4ddf-b681-ef10f00c163c None admin] PATH_INFO: `/v3/domains` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:50.450205 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e5b2946-4ac2-4ddf-b681-ef10f00c163c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:50.450582 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e5b2946-4ac2-4ddf-b681-ef10f00c163c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:50.452088 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e5b2946-4ac2-4ddf-b681-ef10f00c163c None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:50.465275 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 295/594] 10.222.0.36 () {56 vars in 1109 bytes} [Thu Oct 20 08:48:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:50.471011 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e5b2946-4ac2-4ddf-b681-ef10f00c163c None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:50.523888 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:50.594164 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e930a6b9-3c5b-4188-8631-d21cdbbd50cb None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:50.631032 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-995c41d8-4297-4c08-bb17-eaddcf4ec940 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:50.631669 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995c41d8-4297-4c08-bb17-eaddcf4ec940 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:50.632003 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995c41d8-4297-4c08-bb17-eaddcf4ec940 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:50.632278 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995c41d8-4297-4c08-bb17-eaddcf4ec940 None admin] PATH_INFO: `/v3/services` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:50.632878 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-995c41d8-4297-4c08-bb17-eaddcf4ec940 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:50.634244 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-995c41d8-4297-4c08-bb17-eaddcf4ec940 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:50.645431 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 300/595] 10.222.0.36 () {56 vars in 1087 bytes} [Thu Oct 20 08:48:50 2022] GET /identity/v3/services => generated 772 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 08:48:50.686317 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e5b2946-4ac2-4ddf-b681-ef10f00c163c None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:50.725671 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1949aa0a-eb60-42eb-b727-7f32dd035430 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:50.726328 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1949aa0a-eb60-42eb-b727-7f32dd035430 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:50.726675 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1949aa0a-eb60-42eb-b727-7f32dd035430 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:50.726955 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1949aa0a-eb60-42eb-b727-7f32dd035430 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:50.735318 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1949aa0a-eb60-42eb-b727-7f32dd035430 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:50.736889 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1949aa0a-eb60-42eb-b727-7f32dd035430 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:50.750612 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-1949aa0a-eb60-42eb-b727-7f32dd035430 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 20 08:48:50.750612 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:50.750612 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:50.750612 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:50.750612 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:50.750612 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:50.750612 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:50.750612 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:50.750612 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:50.750612 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:50.750612 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:50.750612 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:50.750612 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 20 08:48:50.750612 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 20 08:48:50.750612 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 20 08:48:50.750612 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 20 08:48:50.751650 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:50.751650 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:50.751650 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 20 08:48:50.751650 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 20 08:48:50.751650 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 20 08:48:50.751650 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 20 08:48:50.751650 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:50.751650 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:50.751650 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:50.751650 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:50.751650 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:50.751650 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:50.751650 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:50.751650 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:50.751650 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:50.751650 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:50.751650 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:50.751650 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:50.752628 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:50.752628 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:50.752628 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 20 08:48:50.752628 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 20 08:48:50.752628 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 20 08:48:50.752628 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 20 08:48:50.752628 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 20 08:48:50.752628 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 20 08:48:50.752628 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 20 08:48:50.752628 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:48:50.752628 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 296/596] 10.222.0.36 () {56 vars in 1096 bytes} [Thu Oct 20 08:48:50 2022] GET /identity/v3/users/swift => generated 83 bytes in 281 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:50.759718 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-995c41d8-4297-4c08-bb17-eaddcf4ec940 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:50.902009 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-995c41d8-4297-4c08-bb17-eaddcf4ec940 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:50.904301 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:50.938286 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecd38895-0075-4b70-85bd-8efff27155a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:50.938961 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd38895-0075-4b70-85bd-8efff27155a7 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:50.939231 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd38895-0075-4b70-85bd-8efff27155a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:50.939547 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd38895-0075-4b70-85bd-8efff27155a7 None admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:50.940332 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecd38895-0075-4b70-85bd-8efff27155a7 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:50.940638 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecd38895-0075-4b70-85bd-8efff27155a7 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:50.942055 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecd38895-0075-4b70-85bd-8efff27155a7 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:50.956006 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8fe050-f2a2-4746-92fe-982556a01c66 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:50.956291 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8fe050-f2a2-4746-92fe-982556a01c66 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:50.956569 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8fe050-f2a2-4746-92fe-982556a01c66 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:50.957918 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 297/597] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:48:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:48:50.960577 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 301/598] 10.222.0.36 () {56 vars in 1135 bytes} [Thu Oct 20 08:48:50 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:50.963736 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf6b31ee-6d4d-43c0-9f02-baed3acc084d None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:50.964024 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf6b31ee-6d4d-43c0-9f02-baed3acc084d None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:50.964296 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf6b31ee-6d4d-43c0-9f02-baed3acc084d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:50.966209 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecd38895-0075-4b70-85bd-8efff27155a7 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:50.998665 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-cf6b31ee-6d4d-43c0-9f02-baed3acc084d None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:48:51.000145 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-cf6b31ee-6d4d-43c0-9f02-baed3acc084d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:48:51.013541 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:51.112698 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecd38895-0075-4b70-85bd-8efff27155a7 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:51.124024 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 298/599] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:48:50 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 20 08:48:51.176265 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4301c509-f48f-4158-9d46-cc05d0e2e3c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:51.176790 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4301c509-f48f-4158-9d46-cc05d0e2e3c5 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:51.177098 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4301c509-f48f-4158-9d46-cc05d0e2e3c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:51.177372 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4301c509-f48f-4158-9d46-cc05d0e2e3c5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:51.226342 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:51.277531 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4301c509-f48f-4158-9d46-cc05d0e2e3c5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:51.279019 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4301c509-f48f-4158-9d46-cc05d0e2e3c5 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:51.430299 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 302/600] 10.222.0.36 () {58 vars in 1282 bytes} [Thu Oct 20 08:48:50 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 465 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 20 08:48:51.435535 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf6b31ee-6d4d-43c0-9f02-baed3acc084d None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:51.452594 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4301c509-f48f-4158-9d46-cc05d0e2e3c5 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:51.539552 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:51.559737 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4301c509-f48f-4158-9d46-cc05d0e2e3c5 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:51.596960 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f762aa12-6d9b-4bf2-8a0e-64631a9e47ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:51.597593 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f762aa12-6d9b-4bf2-8a0e-64631a9e47ad None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:51.597872 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f762aa12-6d9b-4bf2-8a0e-64631a9e47ad None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:51.598141 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f762aa12-6d9b-4bf2-8a0e-64631a9e47ad None admin] PATH_INFO: `/v3/roles/service` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:51.609371 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f762aa12-6d9b-4bf2-8a0e-64631a9e47ad None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 20 08:48:51.614579 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-f762aa12-6d9b-4bf2-8a0e-64631a9e47ad None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 20 08:48:51.614579 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:51.614579 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:51.614579 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:51.614579 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:51.614579 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:51.614579 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:51.614579 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:51.614579 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:51.614579 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:51.614579 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:51.614579 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:51.614579 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 20 08:48:51.614579 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 20 08:48:51.614579 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 20 08:48:51.614579 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 20 08:48:51.615605 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 20 08:48:51.615605 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 20 08:48:51.615605 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 20 08:48:51.615605 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 20 08:48:51.615605 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:51.615605 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:51.615605 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:51.615605 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:51.615605 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:51.615605 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:51.615605 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:51.615605 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:51.615605 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:51.615605 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:51.615605 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:51.615605 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:51.616454 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:51.616454 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:51.616454 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:51.616454 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:51.616454 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 20 08:48:51.616454 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 20 08:48:51.616454 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 20 08:48:51.616454 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 20 08:48:51.616454 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 20 08:48:51.616454 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 20 08:48:51.616454 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 20 08:48:51.616454 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:51.616454 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 303/601] 10.222.0.36 () {56 vars in 1102 bytes} [Thu Oct 20 08:48:51 2022] GET /identity/v3/roles/service => generated 85 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 20 08:48:51.621743 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f762aa12-6d9b-4bf2-8a0e-64631a9e47ad None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:51.655249 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf6b31ee-6d4d-43c0-9f02-baed3acc084d None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:51.715570 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f872857-f30d-4cc2-bb7e-7bff776609bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:51.716625 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f872857-f30d-4cc2-bb7e-7bff776609bd None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:51.717106 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f872857-f30d-4cc2-bb7e-7bff776609bd None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:51.717596 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f872857-f30d-4cc2-bb7e-7bff776609bd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:51.728884 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f762aa12-6d9b-4bf2-8a0e-64631a9e47ad None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:51.734163 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f872857-f30d-4cc2-bb7e-7bff776609bd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:51.735674 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f872857-f30d-4cc2-bb7e-7bff776609bd None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:51.744908 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:51.755900 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:51.770481 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-1f872857-f30d-4cc2-bb7e-7bff776609bd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:51.770481 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:51.770481 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:51.770481 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:51.770481 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:51.770481 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:51.770481 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:51.770481 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:51.770481 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:51.770481 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:51.770481 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:51.770481 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:51.770481 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 20 08:48:51.770481 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 20 08:48:51.770481 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 20 08:48:51.770481 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 20 08:48:51.772092 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:51.772092 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:51.772092 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:51.772092 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:51.772092 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:51.772092 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:51.772092 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:51.772092 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:51.772092 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:51.772092 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:51.772092 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:51.772092 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:51.772092 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:51.772092 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:51.772092 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:51.772092 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:51.772092 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 20 08:48:51.773026 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 20 08:48:51.773026 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:51.773026 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:48:51.773996 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-185c06f2-c361-4937-b0f2-3f9685e46df7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:51.778526 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 299/602] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:48:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 337 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 20 08:48:51.779146 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185c06f2-c361-4937-b0f2-3f9685e46df7 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:51.779436 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185c06f2-c361-4937-b0f2-3f9685e46df7 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:51.779852 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185c06f2-c361-4937-b0f2-3f9685e46df7 None admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:51.780616 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-185c06f2-c361-4937-b0f2-3f9685e46df7 None admin] RBAC: Adding query filter params (name=service) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:51.781367 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-185c06f2-c361-4937-b0f2-3f9685e46df7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:51.782389 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f872857-f30d-4cc2-bb7e-7bff776609bd None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:51.783173 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-185c06f2-c361-4937-b0f2-3f9685e46df7 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:51.794145 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 304/603] 10.222.0.36 () {56 vars in 1103 bytes} [Thu Oct 20 08:48:51 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 08:48:51.799961 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-185c06f2-c361-4937-b0f2-3f9685e46df7 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:51.931807 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f872857-f30d-4cc2-bb7e-7bff776609bd None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:51.939795 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-185c06f2-c361-4937-b0f2-3f9685e46df7 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:51.977014 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6964f83-ddfe-4cfb-a4b4-4f8c548b7c19 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:51.977549 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6964f83-ddfe-4cfb-a4b4-4f8c548b7c19 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:51.977831 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6964f83-ddfe-4cfb-a4b4-4f8c548b7c19 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:51.978098 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6964f83-ddfe-4cfb-a4b4-4f8c548b7c19 None admin] PATH_INFO: `/v3/domains` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:51.982719 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6964f83-ddfe-4cfb-a4b4-4f8c548b7c19 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:51.983040 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6964f83-ddfe-4cfb-a4b4-4f8c548b7c19 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:51.984444 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6964f83-ddfe-4cfb-a4b4-4f8c548b7c19 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:52.001681 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9685b302-ef4a-4efa-aed2-5cd88ff1b314 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:52.002317 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9685b302-ef4a-4efa-aed2-5cd88ff1b314 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:52.002630 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9685b302-ef4a-4efa-aed2-5cd88ff1b314 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:52.002903 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9685b302-ef4a-4efa-aed2-5cd88ff1b314 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:52.004336 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 300/604] 10.222.0.36 () {56 vars in 1109 bytes} [Thu Oct 20 08:48:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:52.009334 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6964f83-ddfe-4cfb-a4b4-4f8c548b7c19 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:52.019436 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9685b302-ef4a-4efa-aed2-5cd88ff1b314 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:52.020936 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9685b302-ef4a-4efa-aed2-5cd88ff1b314 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:52.037147 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-9685b302-ef4a-4efa-aed2-5cd88ff1b314 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:52.037147 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:52.037147 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:52.037147 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:52.037147 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:52.037147 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:52.037147 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:52.037147 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:52.037147 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:52.037147 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:52.037147 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:52.037147 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:52.037147 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 20 08:48:52.037147 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 20 08:48:52.037147 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 20 08:48:52.037147 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 20 08:48:52.038117 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:52.038117 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:52.038117 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:52.038117 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:52.038117 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:52.038117 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:52.038117 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:52.038117 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:52.038117 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:52.038117 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:52.038117 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:52.038117 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:52.038117 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:52.038117 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:52.038117 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:52.038117 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:52.038117 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 20 08:48:52.039077 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 20 08:48:52.039077 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:52.039077 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:52.042801 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 305/605] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:48:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 243 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 20 08:48:52.048484 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9685b302-ef4a-4efa-aed2-5cd88ff1b314 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:52.119818 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:52.122545 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:52.197229 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9685b302-ef4a-4efa-aed2-5cd88ff1b314 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:52.222255 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6964f83-ddfe-4cfb-a4b4-4f8c548b7c19 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:52.235218 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56cf8c9d-0b47-4eb3-9947-557a729d40a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:52.235758 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56cf8c9d-0b47-4eb3-9947-557a729d40a1 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:52.236037 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56cf8c9d-0b47-4eb3-9947-557a729d40a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:52.236389 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56cf8c9d-0b47-4eb3-9947-557a729d40a1 None admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:52.237071 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56cf8c9d-0b47-4eb3-9947-557a729d40a1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:52.237375 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56cf8c9d-0b47-4eb3-9947-557a729d40a1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:52.238923 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56cf8c9d-0b47-4eb3-9947-557a729d40a1 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:52.254128 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 306/606] 10.222.0.36 () {56 vars in 1109 bytes} [Thu Oct 20 08:48:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:52.259866 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56cf8c9d-0b47-4eb3-9947-557a729d40a1 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:52.318002 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec4880e-6a50-4bb8-b524-40737036fb87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:52.319043 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec4880e-6a50-4bb8-b524-40737036fb87 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:52.319558 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec4880e-6a50-4bb8-b524-40737036fb87 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:52.320037 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec4880e-6a50-4bb8-b524-40737036fb87 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:52.335570 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fec4880e-6a50-4bb8-b524-40737036fb87 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:52.337047 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fec4880e-6a50-4bb8-b524-40737036fb87 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:52.360267 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-fec4880e-6a50-4bb8-b524-40737036fb87 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 20 08:48:52.360267 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:52.360267 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:52.360267 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:52.360267 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:52.360267 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:52.360267 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:52.360267 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:52.360267 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:52.360267 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:52.360267 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:52.360267 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:52.360267 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 20 08:48:52.360267 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 20 08:48:52.360267 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 20 08:48:52.361314 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 20 08:48:52.361314 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:52.361314 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:52.361314 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:52.361314 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:52.361314 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:52.361314 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:52.361314 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:52.361314 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:52.361314 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:52.361314 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:52.361314 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:52.361314 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:52.361314 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:52.361314 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:52.361314 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:52.361314 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:52.362236 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 20 08:48:52.362236 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 20 08:48:52.362236 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 20 08:48:52.362236 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 20 08:48:52.362236 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 20 08:48:52.362236 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 20 08:48:52.362236 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 20 08:48:52.362236 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:48:52.362236 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 301/607] 10.222.0.36 () {56 vars in 1111 bytes} [Thu Oct 20 08:48:52 2022] GET /identity/v3/projects/service => generated 88 bytes in 353 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 20 08:48:52.370157 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:52.377137 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56cf8c9d-0b47-4eb3-9947-557a729d40a1 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:52.381958 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec4880e-6a50-4bb8-b524-40737036fb87 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:52.424263 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1efb2a46-533f-4e15-b5c8-358e49017c84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:52.424917 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1efb2a46-533f-4e15-b5c8-358e49017c84 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:52.425202 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1efb2a46-533f-4e15-b5c8-358e49017c84 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:52.425567 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1efb2a46-533f-4e15-b5c8-358e49017c84 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:52.435033 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1efb2a46-533f-4e15-b5c8-358e49017c84 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:52.436487 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1efb2a46-533f-4e15-b5c8-358e49017c84 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:52.451105 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-1efb2a46-533f-4e15-b5c8-358e49017c84 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 20 08:48:52.451105 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:52.451105 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:52.451105 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:52.451105 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:52.451105 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:52.451105 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:52.451105 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:52.451105 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:52.451105 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:52.451105 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:52.451105 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:52.451105 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 20 08:48:52.451105 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 20 08:48:52.451105 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 20 08:48:52.451105 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 20 08:48:52.453259 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:52.453259 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:52.453259 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 20 08:48:52.453259 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 20 08:48:52.453259 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 20 08:48:52.453259 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 20 08:48:52.453259 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:52.453259 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:52.453259 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:52.453259 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:52.453259 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:52.453259 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:52.453259 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:52.453259 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:52.453259 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:52.453259 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:52.453259 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:52.453259 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:52.455543 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:52.455543 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:52.455543 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 20 08:48:52.455543 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 20 08:48:52.455543 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 20 08:48:52.455543 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 20 08:48:52.455543 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 20 08:48:52.455543 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 20 08:48:52.455543 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 20 08:48:52.455543 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:52.455543 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 307/608] 10.222.0.36 () {56 vars in 1117 bytes} [Thu Oct 20 08:48:52 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 20 08:48:52.455543 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aebbe6da-60d0-446a-954a-8e303b8202be None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:52.455543 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aebbe6da-60d0-446a-954a-8e303b8202be None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:52.461807 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aebbe6da-60d0-446a-954a-8e303b8202be None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:52.461807 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 308/609] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:48:52 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:48:52.463040 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aebbe6da-60d0-446a-954a-8e303b8202be None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:52.516498 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec4880e-6a50-4bb8-b524-40737036fb87 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:52.532857 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:52.552698 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-538d2e74-d7ef-4464-bdb4-32ad2e4152c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:52.553333 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-538d2e74-d7ef-4464-bdb4-32ad2e4152c7 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:52.553613 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-538d2e74-d7ef-4464-bdb4-32ad2e4152c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:52.553876 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-538d2e74-d7ef-4464-bdb4-32ad2e4152c7 None admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:52.554577 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-538d2e74-d7ef-4464-bdb4-32ad2e4152c7 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:52.554889 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-538d2e74-d7ef-4464-bdb4-32ad2e4152c7 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:52.556299 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-538d2e74-d7ef-4464-bdb4-32ad2e4152c7 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:52.570522 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 302/610] 10.222.0.36 () {56 vars in 1148 bytes} [Thu Oct 20 08:48:52 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:52.570522 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49460210-4a98-44f1-8ea1-5aea4d80ff22 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:52.570522 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49460210-4a98-44f1-8ea1-5aea4d80ff22 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:52.570522 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49460210-4a98-44f1-8ea1-5aea4d80ff22 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:52.585828 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aebbe6da-60d0-446a-954a-8e303b8202be None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:52.589037 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-49460210-4a98-44f1-8ea1-5aea4d80ff22 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:48:52.590249 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-49460210-4a98-44f1-8ea1-5aea4d80ff22 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:48:52.621181 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aea97d52-c85b-41d4-9f10-a52e71bf4552 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:52.621907 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea97d52-c85b-41d4-9f10-a52e71bf4552 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:52.622183 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea97d52-c85b-41d4-9f10-a52e71bf4552 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:52.622462 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea97d52-c85b-41d4-9f10-a52e71bf4552 None admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:52.623190 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aea97d52-c85b-41d4-9f10-a52e71bf4552 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:52.623489 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aea97d52-c85b-41d4-9f10-a52e71bf4552 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:52.624913 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aea97d52-c85b-41d4-9f10-a52e71bf4552 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:52.648264 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 309/611] 10.222.0.36 () {56 vars in 1149 bytes} [Thu Oct 20 08:48:52 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:52.648264 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bade5bef-21c3-451d-ac89-f22b18b49a54 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:52.648264 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bade5bef-21c3-451d-ac89-f22b18b49a54 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:52.648264 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bade5bef-21c3-451d-ac89-f22b18b49a54 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:52.648264 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 310/612] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:48:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:48:52.648264 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bade5bef-21c3-451d-ac89-f22b18b49a54 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:52.665111 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:52.674540 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 303/613] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:48:52 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 20 08:48:52.674540 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49460210-4a98-44f1-8ea1-5aea4d80ff22 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:52.781031 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49460210-4a98-44f1-8ea1-5aea4d80ff22 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:52.791370 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:52.800032 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bade5bef-21c3-451d-ac89-f22b18b49a54 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:52.822007 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e95c04-d37a-4649-9c98-0cc74e3eafb6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:52.822848 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e95c04-d37a-4649-9c98-0cc74e3eafb6 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:52.823614 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e95c04-d37a-4649-9c98-0cc74e3eafb6 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:52.824021 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e95c04-d37a-4649-9c98-0cc74e3eafb6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:52.844967 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2fb95f7-bf7a-41c5-9f1a-b56ba194f6ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:52.846065 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2fb95f7-bf7a-41c5-9f1a-b56ba194f6ec None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:52.846518 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2fb95f7-bf7a-41c5-9f1a-b56ba194f6ec None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:52.846928 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2fb95f7-bf7a-41c5-9f1a-b56ba194f6ec None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:52.848041 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2fb95f7-bf7a-41c5-9f1a-b56ba194f6ec None admin] RBAC: Adding query filter params (role.id=d9d3bc62f6e8453488c74856ef333bb3, scope.project.id=d9857b610fb542969cfb5efbbdb8ce61, user.id=4d558918cd8b4ebea4fe8765297304db) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:52.849498 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2fb95f7-bf7a-41c5-9f1a-b56ba194f6ec None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:52.851108 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2fb95f7-bf7a-41c5-9f1a-b56ba194f6ec None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:52.859629 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 311/614] 10.222.0.36 () {56 vars in 1374 bytes} [Thu Oct 20 08:48:52 2022] GET /identity/v3/role_assignments?user.id=4d558918cd8b4ebea4fe8765297304db&scope.project.id=d9857b610fb542969cfb5efbbdb8ce61&role.id=d9d3bc62f6e8453488c74856ef333bb3 => generated 608 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:52.861766 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6edeaa01-cb99-402e-9112-e6a8618d6a8c None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:52.862453 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6edeaa01-cb99-402e-9112-e6a8618d6a8c None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:52.862950 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6edeaa01-cb99-402e-9112-e6a8618d6a8c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:52.890102 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-6edeaa01-cb99-402e-9112-e6a8618d6a8c None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:48:52.892096 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-6edeaa01-cb99-402e-9112-e6a8618d6a8c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:48:52.912597 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69e95c04-d37a-4649-9c98-0cc74e3eafb6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:52.914714 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69e95c04-d37a-4649-9c98-0cc74e3eafb6 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:53.018861 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 312/615] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:48:52 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 20 08:48:53.019807 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6edeaa01-cb99-402e-9112-e6a8618d6a8c None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:53.031912 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:53.101636 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 304/616] 10.222.0.36 () {58 vars in 1282 bytes} [Thu Oct 20 08:48:52 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 435 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 20 08:48:53.106971 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e95c04-d37a-4649-9c98-0cc74e3eafb6 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:53.118413 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:53.150151 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6edeaa01-cb99-402e-9112-e6a8618d6a8c None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:53.192148 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6367cdf-6bf8-4840-a786-a7cb3ca61123 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:53.192807 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6367cdf-6bf8-4840-a786-a7cb3ca61123 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:53.193080 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6367cdf-6bf8-4840-a786-a7cb3ca61123 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:53.193346 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6367cdf-6bf8-4840-a786-a7cb3ca61123 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:53.194031 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6367cdf-6bf8-4840-a786-a7cb3ca61123 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:53.195511 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6367cdf-6bf8-4840-a786-a7cb3ca61123 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:53.205077 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-d6367cdf-6bf8-4840-a786-a7cb3ca61123 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 20 08:48:53.205077 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:53.205077 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 20 08:48:53.205077 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 20 08:48:53.205077 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 20 08:48:53.205077 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 20 08:48:53.205077 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 20 08:48:53.205077 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 20 08:48:53.205077 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 20 08:48:53.205077 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:53.205077 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 20 08:48:53.205077 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:53.205077 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:53.205077 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:53.205077 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:53.205077 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:53.205077 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:53.206096 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:53.206096 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:53.206096 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:53.206096 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:53.206096 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:53.206096 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:53.206096 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 20 08:48:53.206096 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 20 08:48:53.206096 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 20 08:48:53.206096 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 20 08:48:53.206096 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:53.206096 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:53.206096 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:53.206096 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:53.206096 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:53.206096 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:53.207143 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:53.207143 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:53.207143 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:53.207143 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:53.207143 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:53.207143 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:53.207143 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:53.207143 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:53.207143 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:53.207143 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:53.207143 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 20 08:48:53.207143 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 20 08:48:53.207143 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 20 08:48:53.207143 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:53.207143 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 313/617] 10.222.0.36 () {56 vars in 1129 bytes} [Thu Oct 20 08:48:53 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:53.213081 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6367cdf-6bf8-4840-a786-a7cb3ca61123 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:53.233130 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e95c04-d37a-4649-9c98-0cc74e3eafb6 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:53.277791 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5778202e-f4c0-472d-bec8-c2a43610ddef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:53.279029 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5778202e-f4c0-472d-bec8-c2a43610ddef None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:53.279436 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5778202e-f4c0-472d-bec8-c2a43610ddef None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:53.279708 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5778202e-f4c0-472d-bec8-c2a43610ddef None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:53.294695 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5778202e-f4c0-472d-bec8-c2a43610ddef None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:53.295334 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:53.296129 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5778202e-f4c0-472d-bec8-c2a43610ddef None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:53.318775 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-5778202e-f4c0-472d-bec8-c2a43610ddef None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:53.318775 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:53.318775 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:53.318775 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:53.318775 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:53.318775 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:53.318775 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:53.318775 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:53.318775 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:53.318775 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:53.318775 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:53.318775 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:53.318775 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 20 08:48:53.318775 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 20 08:48:53.318775 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 20 08:48:53.318775 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 20 08:48:53.319908 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:53.319908 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:53.319908 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:53.319908 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:53.319908 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:53.319908 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:53.319908 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:53.319908 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:53.319908 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:53.319908 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:53.319908 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:53.319908 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:53.319908 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:53.319908 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:53.319908 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:53.319908 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:53.319908 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 20 08:48:53.320808 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 20 08:48:53.320808 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:53.320808 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:48:53.323577 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 305/618] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:48:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 20 08:48:53.326102 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b748f81-6a29-4858-ad77-33f65a404f64 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:53.326710 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b748f81-6a29-4858-ad77-33f65a404f64 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:53.327528 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b748f81-6a29-4858-ad77-33f65a404f64 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:53.330690 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 306/619] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:48:53 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:48:53.331741 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b748f81-6a29-4858-ad77-33f65a404f64 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:53.342826 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6367cdf-6bf8-4840-a786-a7cb3ca61123 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:53.379288 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6da4475-7d50-4e55-a3cd-ad50cab38b93 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:53.380187 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6da4475-7d50-4e55-a3cd-ad50cab38b93 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:53.380773 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6da4475-7d50-4e55-a3cd-ad50cab38b93 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:53.381207 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6da4475-7d50-4e55-a3cd-ad50cab38b93 None admin] PATH_INFO: `/v3/services` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:53.381837 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6da4475-7d50-4e55-a3cd-ad50cab38b93 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:53.382134 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6da4475-7d50-4e55-a3cd-ad50cab38b93 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:53.383524 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6da4475-7d50-4e55-a3cd-ad50cab38b93 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:53.392740 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 314/620] 10.222.0.36 () {56 vars in 1124 bytes} [Thu Oct 20 08:48:53 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:53.392740 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04020677-4ebc-4198-a6be-a037f3212bfa None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:53.392740 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04020677-4ebc-4198-a6be-a037f3212bfa None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:53.392740 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04020677-4ebc-4198-a6be-a037f3212bfa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:53.408332 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-04020677-4ebc-4198-a6be-a037f3212bfa None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:48:53.409565 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-04020677-4ebc-4198-a6be-a037f3212bfa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:48:53.413557 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:53.461075 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b748f81-6a29-4858-ad77-33f65a404f64 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:53.481763 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 315/621] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:48:53 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 20 08:48:53.483103 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04020677-4ebc-4198-a6be-a037f3212bfa None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:53.512455 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dd3a158-40ab-4150-8e3d-918db2c4d5d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:53.513993 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd3a158-40ab-4150-8e3d-918db2c4d5d8 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:53.514492 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd3a158-40ab-4150-8e3d-918db2c4d5d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:53.514954 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd3a158-40ab-4150-8e3d-918db2c4d5d8 None admin] PATH_INFO: `/v3/domains` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:53.515856 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dd3a158-40ab-4150-8e3d-918db2c4d5d8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:53.516583 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dd3a158-40ab-4150-8e3d-918db2c4d5d8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:53.519079 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dd3a158-40ab-4150-8e3d-918db2c4d5d8 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:53.535337 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 307/622] 10.222.0.36 () {56 vars in 1109 bytes} [Thu Oct 20 08:48:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:53.537071 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dd3a158-40ab-4150-8e3d-918db2c4d5d8 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:53.551890 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:53.590573 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04020677-4ebc-4198-a6be-a037f3212bfa None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:53.627174 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-511e871f-c356-4a99-bc9a-dac9214a3e9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:53.627902 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-511e871f-c356-4a99-bc9a-dac9214a3e9f None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:53.628164 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-511e871f-c356-4a99-bc9a-dac9214a3e9f None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:53.628543 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-511e871f-c356-4a99-bc9a-dac9214a3e9f None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:53.629183 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-511e871f-c356-4a99-bc9a-dac9214a3e9f None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:53.631499 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-511e871f-c356-4a99-bc9a-dac9214a3e9f None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:53.642331 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-511e871f-c356-4a99-bc9a-dac9214a3e9f None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 20 08:48:53.642331 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:53.642331 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 20 08:48:53.642331 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 20 08:48:53.642331 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 20 08:48:53.642331 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 20 08:48:53.642331 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 20 08:48:53.642331 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 20 08:48:53.642331 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 20 08:48:53.642331 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:53.642331 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 20 08:48:53.642331 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:53.642331 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:53.642331 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:53.642331 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:53.642331 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:53.643615 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:53.643615 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:53.643615 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:53.643615 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:53.643615 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:53.643615 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:53.643615 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:53.643615 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 20 08:48:53.643615 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 20 08:48:53.643615 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 20 08:48:53.643615 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 20 08:48:53.643615 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:53.643615 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:53.643615 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:53.643615 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:53.643615 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:53.643615 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:53.644803 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:53.644803 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:53.644803 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:53.644803 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:53.644803 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:53.644803 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:53.644803 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:53.644803 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:53.644803 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:53.644803 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:53.644803 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 20 08:48:53.644803 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 20 08:48:53.644803 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 20 08:48:53.644803 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:53.650831 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 316/623] 10.222.0.36 () {56 vars in 1132 bytes} [Thu Oct 20 08:48:53 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:53.651786 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-511e871f-c356-4a99-bc9a-dac9214a3e9f None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:53.672118 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dd3a158-40ab-4150-8e3d-918db2c4d5d8 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:53.695240 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:53.721684 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9955d51c-dca3-4e10-b68b-ed23649a19dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:53.723199 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9955d51c-dca3-4e10-b68b-ed23649a19dd None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:53.723642 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9955d51c-dca3-4e10-b68b-ed23649a19dd None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:53.724095 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9955d51c-dca3-4e10-b68b-ed23649a19dd None admin] PATH_INFO: `/v3/services` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:53.724876 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9955d51c-dca3-4e10-b68b-ed23649a19dd None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:53.725329 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9955d51c-dca3-4e10-b68b-ed23649a19dd None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:53.727126 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9955d51c-dca3-4e10-b68b-ed23649a19dd None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:53.734786 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 308/624] 10.222.0.36 () {56 vars in 1124 bytes} [Thu Oct 20 08:48:53 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:53.735843 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9955d51c-dca3-4e10-b68b-ed23649a19dd None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:53.769965 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-511e871f-c356-4a99-bc9a-dac9214a3e9f None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:53.801175 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:53.809940 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7719b405-03c4-4e86-a636-f82df1085a68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:53.810804 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7719b405-03c4-4e86-a636-f82df1085a68 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:53.811089 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7719b405-03c4-4e86-a636-f82df1085a68 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:53.811359 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7719b405-03c4-4e86-a636-f82df1085a68 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:53.819004 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7719b405-03c4-4e86-a636-f82df1085a68 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:53.820637 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7719b405-03c4-4e86-a636-f82df1085a68 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:53.832511 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-7719b405-03c4-4e86-a636-f82df1085a68 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 20 08:48:53.832511 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:53.832511 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:53.832511 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:53.832511 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:53.832511 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:53.832511 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:53.832511 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:53.832511 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:53.832511 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:53.832511 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:53.832511 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:53.832511 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 20 08:48:53.832511 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 20 08:48:53.832511 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 20 08:48:53.834303 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 20 08:48:53.834303 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:53.834303 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:53.834303 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:53.834303 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:53.834303 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:53.834303 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:53.834303 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:53.834303 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:53.834303 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:53.834303 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:53.834303 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:53.834303 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:53.834303 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:53.834303 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:53.834303 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:53.834303 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:53.835660 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 20 08:48:53.835660 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 20 08:48:53.835660 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 20 08:48:53.835660 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 20 08:48:53.835660 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 20 08:48:53.835660 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 20 08:48:53.835660 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 20 08:48:53.835660 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:53.835660 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 317/625] 10.222.0.36 () {56 vars in 1111 bytes} [Thu Oct 20 08:48:53 2022] GET /identity/v3/projects/service => generated 88 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:53.835660 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7719b405-03c4-4e86-a636-f82df1085a68 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:53.849943 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9955d51c-dca3-4e10-b68b-ed23649a19dd None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:53.883474 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a17075f9-57f5-45b0-96df-839851908a4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:53.884071 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a17075f9-57f5-45b0-96df-839851908a4e None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:53.884342 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a17075f9-57f5-45b0-96df-839851908a4e None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:53.884608 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a17075f9-57f5-45b0-96df-839851908a4e None admin] PATH_INFO: `/v3/services` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:53.885223 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a17075f9-57f5-45b0-96df-839851908a4e None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:53.885518 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a17075f9-57f5-45b0-96df-839851908a4e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:53.886917 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a17075f9-57f5-45b0-96df-839851908a4e None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:53.896269 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 309/626] 10.222.0.36 () {56 vars in 1126 bytes} [Thu Oct 20 08:48:53 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:53.896269 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a17075f9-57f5-45b0-96df-839851908a4e None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:53.948217 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:53.965001 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7719b405-03c4-4e86-a636-f82df1085a68 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:53.999814 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7701bf0-d67a-4b36-a2b2-6fd043e7052c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:54.000425 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7701bf0-d67a-4b36-a2b2-6fd043e7052c None admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:54.000708 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7701bf0-d67a-4b36-a2b2-6fd043e7052c None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:54.000981 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7701bf0-d67a-4b36-a2b2-6fd043e7052c None admin] PATH_INFO: `/v3/endpoints` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:54.001637 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7701bf0-d67a-4b36-a2b2-6fd043e7052c None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:54.003042 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7701bf0-d67a-4b36-a2b2-6fd043e7052c None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:54.010749 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a17075f9-57f5-45b0-96df-839851908a4e None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:54.028668 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 318/627] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 08:48:53 2022] POST /identity/v3/endpoints => generated 346 bytes in 194 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 08:48:54.029478 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7701bf0-d67a-4b36-a2b2-6fd043e7052c None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:54.067134 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4164b706-ee6b-4be1-bb79-1880b6ffaee9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:54.068348 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4164b706-ee6b-4be1-bb79-1880b6ffaee9 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:54.068694 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4164b706-ee6b-4be1-bb79-1880b6ffaee9 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:54.069022 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4164b706-ee6b-4be1-bb79-1880b6ffaee9 None admin] PATH_INFO: `/v3/services/network` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:54.069735 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4164b706-ee6b-4be1-bb79-1880b6ffaee9 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:54.071493 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4164b706-ee6b-4be1-bb79-1880b6ffaee9 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:54.085233 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-4164b706-ee6b-4be1-bb79-1880b6ffaee9 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Oct 20 08:48:54.085233 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:54.085233 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 20 08:48:54.085233 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 20 08:48:54.085233 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 20 08:48:54.085233 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 20 08:48:54.085233 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 20 08:48:54.085233 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 20 08:48:54.085233 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 20 08:48:54.085233 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:48:54.085233 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 20 08:48:54.085233 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:48:54.085233 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:54.085233 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:54.085233 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:54.085233 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:54.085233 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:54.086646 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:54.086646 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:54.086646 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:54.086646 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:54.086646 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:54.086646 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:54.086646 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 20 08:48:54.086646 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 20 08:48:54.086646 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 20 08:48:54.086646 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 20 08:48:54.086646 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:54.086646 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:54.086646 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:54.086646 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:54.086646 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:54.086646 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:54.087722 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:54.087722 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:54.087722 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:54.087722 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:54.087722 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:54.087722 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:54.087722 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:54.087722 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:54.087722 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:54.087722 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:54.087722 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 20 08:48:54.087722 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 20 08:48:54.087722 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 20 08:48:54.087722 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:48:54.091746 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 310/628] 10.222.0.36 () {56 vars in 1111 bytes} [Thu Oct 20 08:48:53 2022] GET /identity/v3/services/network => generated 88 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:54.092556 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4164b706-ee6b-4be1-bb79-1880b6ffaee9 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:54.104212 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:54.173498 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7701bf0-d67a-4b36-a2b2-6fd043e7052c None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:54.219397 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5024fa86-db58-4924-80f1-d826ffc91ec8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:54.220404 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5024fa86-db58-4924-80f1-d826ffc91ec8 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:54.220893 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5024fa86-db58-4924-80f1-d826ffc91ec8 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:54.221458 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5024fa86-db58-4924-80f1-d826ffc91ec8 None admin] PATH_INFO: `/v3/projects` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:54.222434 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5024fa86-db58-4924-80f1-d826ffc91ec8 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:54.223011 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5024fa86-db58-4924-80f1-d826ffc91ec8 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:54.225168 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5024fa86-db58-4924-80f1-d826ffc91ec8 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:54.229391 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4164b706-ee6b-4be1-bb79-1880b6ffaee9 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:54.240120 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:54.243900 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 319/629] 10.222.0.36 () {56 vars in 1148 bytes} [Thu Oct 20 08:48:54 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:54.245181 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5024fa86-db58-4924-80f1-d826ffc91ec8 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:54.276653 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8368edb5-e856-4f5a-a9e2-9339cc2a4517 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:54.277325 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8368edb5-e856-4f5a-a9e2-9339cc2a4517 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:54.277599 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8368edb5-e856-4f5a-a9e2-9339cc2a4517 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:54.277870 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8368edb5-e856-4f5a-a9e2-9339cc2a4517 None admin] PATH_INFO: `/v3/services` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:54.280356 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8368edb5-e856-4f5a-a9e2-9339cc2a4517 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:54.280356 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8368edb5-e856-4f5a-a9e2-9339cc2a4517 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:54.281097 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8368edb5-e856-4f5a-a9e2-9339cc2a4517 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:54.288068 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 311/630] 10.222.0.36 () {56 vars in 1126 bytes} [Thu Oct 20 08:48:54 2022] GET /identity/v3/services?type=compute_legacy => generated 131 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:54.288959 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8368edb5-e856-4f5a-a9e2-9339cc2a4517 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:54.370903 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5024fa86-db58-4924-80f1-d826ffc91ec8 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:54.382240 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:54.414874 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89c187a0-48a4-447a-804c-8e8e320e5aff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:54.415483 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c187a0-48a4-447a-804c-8e8e320e5aff None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:54.415763 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c187a0-48a4-447a-804c-8e8e320e5aff None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:54.416033 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c187a0-48a4-447a-804c-8e8e320e5aff None admin] PATH_INFO: `/v3/services` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:54.416676 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89c187a0-48a4-447a-804c-8e8e320e5aff None admin] RBAC: Adding query filter params (name=network) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:54.416973 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89c187a0-48a4-447a-804c-8e8e320e5aff None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:54.418351 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89c187a0-48a4-447a-804c-8e8e320e5aff None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:54.423567 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 320/631] 10.222.0.36 () {56 vars in 1112 bytes} [Thu Oct 20 08:48:54 2022] GET /identity/v3/services?name=network => generated 124 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:54.429752 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89c187a0-48a4-447a-804c-8e8e320e5aff None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:54.435517 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8368edb5-e856-4f5a-a9e2-9339cc2a4517 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:54.495262 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdadb5f5-fdde-4daf-ab16-29a79975431f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:54.495872 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdadb5f5-fdde-4daf-ab16-29a79975431f None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:54.496157 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdadb5f5-fdde-4daf-ab16-29a79975431f None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:54.496417 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdadb5f5-fdde-4daf-ab16-29a79975431f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:54.497153 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdadb5f5-fdde-4daf-ab16-29a79975431f None admin] RBAC: Adding query filter params (role.id=f81100d13381464aa48cbae3561ca1bc, scope.project.id=d9857b610fb542969cfb5efbbdb8ce61, user.id=f4b0e9187ce74f7888e9e7094be5c824) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:54.497457 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdadb5f5-fdde-4daf-ab16-29a79975431f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:54.498976 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdadb5f5-fdde-4daf-ab16-29a79975431f None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:54.511810 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 312/632] 10.222.0.36 () {56 vars in 1374 bytes} [Thu Oct 20 08:48:54 2022] GET /identity/v3/role_assignments?user.id=f4b0e9187ce74f7888e9e7094be5c824&scope.project.id=d9857b610fb542969cfb5efbbdb8ce61&role.id=f81100d13381464aa48cbae3561ca1bc => generated 608 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:54.570454 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89c187a0-48a4-447a-804c-8e8e320e5aff None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:54.608846 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3998bacc-66e7-49d8-9787-d4f0d2fe31f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:54.609464 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3998bacc-66e7-49d8-9787-d4f0d2fe31f8 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:54.609738 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3998bacc-66e7-49d8-9787-d4f0d2fe31f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:54.610006 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3998bacc-66e7-49d8-9787-d4f0d2fe31f8 None admin] PATH_INFO: `/v3/services` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:54.610679 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3998bacc-66e7-49d8-9787-d4f0d2fe31f8 None admin] RBAC: Adding query filter params (type=network) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:54.610983 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3998bacc-66e7-49d8-9787-d4f0d2fe31f8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:54.612377 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3998bacc-66e7-49d8-9787-d4f0d2fe31f8 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:54.619028 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 321/633] 10.222.0.36 () {56 vars in 1112 bytes} [Thu Oct 20 08:48:54 2022] GET /identity/v3/services?type=network => generated 350 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:54.624976 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdadb5f5-fdde-4daf-ab16-29a79975431f None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:54.665571 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:54.763046 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdadb5f5-fdde-4daf-ab16-29a79975431f None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:54.799883 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d59944aa-66b3-4232-9865-016f5de87a0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:54.800492 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d59944aa-66b3-4232-9865-016f5de87a0f None admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:54.800771 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d59944aa-66b3-4232-9865-016f5de87a0f None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:54.801042 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d59944aa-66b3-4232-9865-016f5de87a0f None admin] PATH_INFO: `/v3/endpoints` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:54.801710 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d59944aa-66b3-4232-9865-016f5de87a0f None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:54.803748 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d59944aa-66b3-4232-9865-016f5de87a0f None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:54.828275 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 313/634] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 08:48:54 2022] POST /identity/v3/endpoints => generated 337 bytes in 204 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 08:48:55.323588 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ee726b-d964-4747-bdf1-11e239cc77bd None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:55.324006 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ee726b-d964-4747-bdf1-11e239cc77bd None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:55.324120 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ee726b-d964-4747-bdf1-11e239cc77bd None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:55.325696 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 322/635] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:48:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:48:55.331307 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e31671-7259-4919-abcb-88df7f45e703 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:55.331590 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e31671-7259-4919-abcb-88df7f45e703 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:55.331860 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e31671-7259-4919-abcb-88df7f45e703 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:55.351545 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-41e31671-7259-4919-abcb-88df7f45e703 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:48:55.352724 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-41e31671-7259-4919-abcb-88df7f45e703 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:48:55.444800 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 314/636] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:48:55 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 20 08:48:55.712568 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74ee726b-d964-4747-bdf1-11e239cc77bd None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:55.748757 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:55.824933 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74ee726b-d964-4747-bdf1-11e239cc77bd None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:55.860204 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae5fbb1b-d98f-4931-9abb-357b02da2b9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:55.861198 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5fbb1b-d98f-4931-9abb-357b02da2b9d None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:55.861694 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5fbb1b-d98f-4931-9abb-357b02da2b9d None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:55.862182 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5fbb1b-d98f-4931-9abb-357b02da2b9d None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:55.863000 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae5fbb1b-d98f-4931-9abb-357b02da2b9d None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:55.865158 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae5fbb1b-d98f-4931-9abb-357b02da2b9d None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:55.874484 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-ae5fbb1b-d98f-4931-9abb-357b02da2b9d None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 20 08:48:55.874484 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:55.874484 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 20 08:48:55.874484 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 20 08:48:55.874484 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 20 08:48:55.874484 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 20 08:48:55.874484 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 20 08:48:55.874484 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 20 08:48:55.874484 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 20 08:48:55.874484 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:55.874484 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 20 08:48:55.874484 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:55.874484 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:55.874484 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:55.874484 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:55.874484 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:55.874484 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:55.875495 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:55.875495 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:55.875495 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:55.875495 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:55.875495 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:55.875495 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:55.875495 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 20 08:48:55.875495 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 20 08:48:55.875495 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 20 08:48:55.875495 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 20 08:48:55.875495 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:55.875495 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:55.875495 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:55.875495 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:55.875495 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:55.875495 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:55.876352 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:55.876352 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:55.876352 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:55.876352 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:55.876352 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:55.876352 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:55.876352 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:55.876352 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:55.876352 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:55.876352 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:55.876352 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 20 08:48:55.876352 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 20 08:48:55.876352 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 20 08:48:55.876352 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:55.879065 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 323/637] 10.222.0.36 () {56 vars in 1126 bytes} [Thu Oct 20 08:48:55 2022] GET /identity/v3/services/object-store => generated 93 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:55.886520 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41e31671-7259-4919-abcb-88df7f45e703 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:55.902955 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:55.992579 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41e31671-7259-4919-abcb-88df7f45e703 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:56.027510 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e7196bb-7d11-46ad-9573-968504538d51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:56.028120 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7196bb-7d11-46ad-9573-968504538d51 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:56.028460 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7196bb-7d11-46ad-9573-968504538d51 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:56.028735 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7196bb-7d11-46ad-9573-968504538d51 None admin] PATH_INFO: `/v3/services` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:56.029367 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e7196bb-7d11-46ad-9573-968504538d51 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:56.029669 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e7196bb-7d11-46ad-9573-968504538d51 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:56.031078 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e7196bb-7d11-46ad-9573-968504538d51 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:56.036786 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 315/638] 10.222.0.36 () {56 vars in 1122 bytes} [Thu Oct 20 08:48:55 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:56.043742 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae5fbb1b-d98f-4931-9abb-357b02da2b9d None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:56.150835 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:56.154563 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae5fbb1b-d98f-4931-9abb-357b02da2b9d None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:56.195534 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f069ef96-a8d1-45dc-a025-910cfa997980 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:56.196141 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f069ef96-a8d1-45dc-a025-910cfa997980 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:56.196418 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f069ef96-a8d1-45dc-a025-910cfa997980 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:56.196686 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f069ef96-a8d1-45dc-a025-910cfa997980 None admin] PATH_INFO: `/v3/services` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:56.197313 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f069ef96-a8d1-45dc-a025-910cfa997980 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:56.197609 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f069ef96-a8d1-45dc-a025-910cfa997980 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:56.199116 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f069ef96-a8d1-45dc-a025-910cfa997980 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:56.204406 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 324/639] 10.222.0.36 () {56 vars in 1122 bytes} [Thu Oct 20 08:48:56 2022] GET /identity/v3/services?type=object-store => generated 129 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:56.465949 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c34ed8e-9d40-4931-82e2-983d0ea4b2a2 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:56.466297 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c34ed8e-9d40-4931-82e2-983d0ea4b2a2 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:56.466517 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c34ed8e-9d40-4931-82e2-983d0ea4b2a2 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:56.468113 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 316/640] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:48:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:48:56.473339 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8025738a-bdc6-439d-8ef1-6e5f4f77f73e None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:56.473625 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8025738a-bdc6-439d-8ef1-6e5f4f77f73e None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:56.474530 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8025738a-bdc6-439d-8ef1-6e5f4f77f73e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:56.492675 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-8025738a-bdc6-439d-8ef1-6e5f4f77f73e None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:48:56.493853 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-8025738a-bdc6-439d-8ef1-6e5f4f77f73e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:48:56.561418 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 325/641] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:48:56 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 20 08:48:56.825855 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4f897f-59dd-4d57-803f-cbe209adce9e None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:56.826046 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4f897f-59dd-4d57-803f-cbe209adce9e None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:56.826312 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4f897f-59dd-4d57-803f-cbe209adce9e None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:56.827803 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 317/642] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:48:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:48:56.833227 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7bb052e-0ace-44cb-985e-9a7c58ce9732 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:56.833504 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7bb052e-0ace-44cb-985e-9a7c58ce9732 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:56.833775 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7bb052e-0ace-44cb-985e-9a7c58ce9732 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:56.834819 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8025738a-bdc6-439d-8ef1-6e5f4f77f73e None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:56.852226 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-f7bb052e-0ace-44cb-985e-9a7c58ce9732 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:48:56.853389 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-f7bb052e-0ace-44cb-985e-9a7c58ce9732 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:48:56.935855 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:56.937544 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 318/643] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:48:56 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 20 08:48:56.952265 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8025738a-bdc6-439d-8ef1-6e5f4f77f73e None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:56.990549 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0700790d-09d9-4dce-aef7-b2b6d8adfa67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:56.991208 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0700790d-09d9-4dce-aef7-b2b6d8adfa67 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:56.991482 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0700790d-09d9-4dce-aef7-b2b6d8adfa67 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:56.991749 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0700790d-09d9-4dce-aef7-b2b6d8adfa67 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:56.992413 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0700790d-09d9-4dce-aef7-b2b6d8adfa67 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:56.993825 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0700790d-09d9-4dce-aef7-b2b6d8adfa67 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:57.003855 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-0700790d-09d9-4dce-aef7-b2b6d8adfa67 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 20 08:48:57.003855 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:57.003855 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 20 08:48:57.003855 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 20 08:48:57.003855 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 20 08:48:57.003855 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 20 08:48:57.003855 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 20 08:48:57.003855 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 20 08:48:57.003855 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 20 08:48:57.003855 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:57.003855 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 20 08:48:57.003855 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:57.003855 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:57.003855 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:57.003855 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:57.003855 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:57.003855 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:57.004933 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:57.004933 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:57.004933 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:57.004933 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:57.004933 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:57.004933 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:57.004933 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 20 08:48:57.004933 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 20 08:48:57.004933 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 20 08:48:57.004933 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 20 08:48:57.004933 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:57.004933 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:57.004933 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:57.004933 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:57.004933 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:57.004933 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:57.007030 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:57.007030 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:57.007030 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:57.007030 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:57.007030 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:57.007030 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:57.007030 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:57.007030 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:57.007030 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:57.007030 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:57.007030 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 20 08:48:57.007030 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 20 08:48:57.007030 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 20 08:48:57.007030 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:57.007030 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 326/644] 10.222.0.36 () {56 vars in 1114 bytes} [Thu Oct 20 08:48:56 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:57.011893 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7bb052e-0ace-44cb-985e-9a7c58ce9732 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:57.028736 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:57.099175 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae7e228-f427-4511-8f58-b25271e5c7f0 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:57.099655 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae7e228-f427-4511-8f58-b25271e5c7f0 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:57.099933 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae7e228-f427-4511-8f58-b25271e5c7f0 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:57.101689 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 327/645] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:48:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 20 08:48:57.106800 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f9e9288-3278-4343-8d3b-07514bc84bc3 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:57.107079 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f9e9288-3278-4343-8d3b-07514bc84bc3 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:57.107351 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f9e9288-3278-4343-8d3b-07514bc84bc3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:57.125361 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7bb052e-0ace-44cb-985e-9a7c58ce9732 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:57.129474 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-8f9e9288-3278-4343-8d3b-07514bc84bc3 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:48:57.131420 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-8f9e9288-3278-4343-8d3b-07514bc84bc3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:48:57.170449 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-192ac4a6-ea9d-4007-a40d-2ff0a2a76e0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:57.171078 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192ac4a6-ea9d-4007-a40d-2ff0a2a76e0d None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:57.171350 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192ac4a6-ea9d-4007-a40d-2ff0a2a76e0d None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:57.171615 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192ac4a6-ea9d-4007-a40d-2ff0a2a76e0d None admin] PATH_INFO: `/v3/services` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:57.173609 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-192ac4a6-ea9d-4007-a40d-2ff0a2a76e0d None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:57.173916 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-192ac4a6-ea9d-4007-a40d-2ff0a2a76e0d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:57.175304 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-192ac4a6-ea9d-4007-a40d-2ff0a2a76e0d None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:57.180710 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 319/646] 10.222.0.36 () {56 vars in 1114 bytes} [Thu Oct 20 08:48:57 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:57.186213 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-192ac4a6-ea9d-4007-a40d-2ff0a2a76e0d None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:57.215530 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 328/647] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:48:57 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 20 08:48:57.232926 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f9e9288-3278-4343-8d3b-07514bc84bc3 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:57.274523 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:57.306043 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-192ac4a6-ea9d-4007-a40d-2ff0a2a76e0d None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:57.338877 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:57.348706 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f9e9288-3278-4343-8d3b-07514bc84bc3 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:57.351063 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fffcc820-f451-4fd5-880f-a48451bc4206 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:57.352606 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fffcc820-f451-4fd5-880f-a48451bc4206 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:57.353048 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fffcc820-f451-4fd5-880f-a48451bc4206 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:57.353458 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fffcc820-f451-4fd5-880f-a48451bc4206 None admin] PATH_INFO: `/v3/services` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:57.354224 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fffcc820-f451-4fd5-880f-a48451bc4206 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:57.354703 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fffcc820-f451-4fd5-880f-a48451bc4206 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:57.356211 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fffcc820-f451-4fd5-880f-a48451bc4206 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:57.362793 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 320/648] 10.222.0.36 () {56 vars in 1114 bytes} [Thu Oct 20 08:48:57 2022] GET /identity/v3/services?type=volumev3 => generated 125 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:57.391108 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f8e218b-3fd3-4c69-bd2e-1353d0a009da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:57.392058 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8e218b-3fd3-4c69-bd2e-1353d0a009da None admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:57.393083 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8e218b-3fd3-4c69-bd2e-1353d0a009da None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:57.393551 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8e218b-3fd3-4c69-bd2e-1353d0a009da None admin] PATH_INFO: `/v3/services` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:57.394449 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f8e218b-3fd3-4c69-bd2e-1353d0a009da None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:57.396901 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f8e218b-3fd3-4c69-bd2e-1353d0a009da None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:57.422668 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 329/649] 10.222.0.36 () {60 vars in 1141 bytes} [Thu Oct 20 08:48:57 2022] POST /identity/v3/services => generated 269 bytes in 186 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 20 08:48:57.500634 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fffcc820-f451-4fd5-880f-a48451bc4206 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:57.627151 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fffcc820-f451-4fd5-880f-a48451bc4206 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:57.668885 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f3b230e-88d2-4000-a783-044e7bfffa4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:57.669532 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3b230e-88d2-4000-a783-044e7bfffa4b None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:57.669809 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3b230e-88d2-4000-a783-044e7bfffa4b None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:57.670098 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3b230e-88d2-4000-a783-044e7bfffa4b None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:57.672034 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:57.683143 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f3b230e-88d2-4000-a783-044e7bfffa4b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 20 08:48:57.687665 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-3f3b230e-88d2-4000-a783-044e7bfffa4b None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 20 08:48:57.687665 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:57.687665 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:57.687665 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:57.687665 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:57.687665 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:57.687665 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:57.687665 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:57.687665 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:57.687665 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:57.687665 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:57.687665 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:57.687665 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 20 08:48:57.687665 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 20 08:48:57.687665 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 20 08:48:57.687665 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 20 08:48:57.688627 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 20 08:48:57.688627 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 20 08:48:57.688627 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 20 08:48:57.688627 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 20 08:48:57.688627 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:57.688627 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:57.688627 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:57.688627 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:57.688627 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:57.688627 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:57.688627 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:57.688627 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:57.688627 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:57.688627 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:57.688627 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:57.688627 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:57.689996 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:57.689996 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:57.689996 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:57.689996 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:57.689996 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 20 08:48:57.689996 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 20 08:48:57.689996 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 20 08:48:57.689996 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 20 08:48:57.689996 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 20 08:48:57.689996 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 20 08:48:57.689996 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 20 08:48:57.689996 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:48:57.689996 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 321/650] 10.222.0.36 () {56 vars in 1120 bytes} [Thu Oct 20 08:48:57 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:57.695019 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f8e218b-3fd3-4c69-bd2e-1353d0a009da None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:57.798781 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f8e218b-3fd3-4c69-bd2e-1353d0a009da None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:57.838077 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e85b4e9d-5e04-48fe-86ed-f3e970b32317 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:57.838728 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e85b4e9d-5e04-48fe-86ed-f3e970b32317 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:57.838999 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e85b4e9d-5e04-48fe-86ed-f3e970b32317 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:57.839227 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e85b4e9d-5e04-48fe-86ed-f3e970b32317 None admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:57.839886 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e85b4e9d-5e04-48fe-86ed-f3e970b32317 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:57.840179 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e85b4e9d-5e04-48fe-86ed-f3e970b32317 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:57.841537 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e85b4e9d-5e04-48fe-86ed-f3e970b32317 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:57.850922 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 330/651] 10.222.0.36 () {56 vars in 1115 bytes} [Thu Oct 20 08:48:57 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:57.859746 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f3b230e-88d2-4000-a783-044e7bfffa4b None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:57.967560 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f3b230e-88d2-4000-a783-044e7bfffa4b None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:58.003594 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aafa33b8-8c7a-4e02-ae0c-7788b2234c0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:58.004220 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aafa33b8-8c7a-4e02-ae0c-7788b2234c0a None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:58.004493 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aafa33b8-8c7a-4e02-ae0c-7788b2234c0a None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:58.004757 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aafa33b8-8c7a-4e02-ae0c-7788b2234c0a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:58.016005 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aafa33b8-8c7a-4e02-ae0c-7788b2234c0a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:58.017535 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aafa33b8-8c7a-4e02-ae0c-7788b2234c0a None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:58.032572 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-aafa33b8-8c7a-4e02-ae0c-7788b2234c0a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:58.032572 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:58.032572 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:58.032572 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:58.032572 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:58.032572 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:58.032572 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:58.032572 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:58.032572 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:58.032572 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:58.032572 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:58.032572 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:58.032572 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 20 08:48:58.032572 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 20 08:48:58.032572 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 20 08:48:58.032572 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 20 08:48:58.033626 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:58.033626 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:58.033626 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:58.033626 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:58.033626 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:58.033626 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:58.033626 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:58.033626 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:58.033626 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:58.033626 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:58.033626 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:58.033626 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:58.033626 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:58.033626 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:58.033626 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:58.033626 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:58.033626 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 20 08:48:58.034571 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 20 08:48:58.034571 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:58.034571 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:48:58.034571 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 322/652] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:48:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:58.043542 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e85b4e9d-5e04-48fe-86ed-f3e970b32317 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:58.058823 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:58.155539 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e85b4e9d-5e04-48fe-86ed-f3e970b32317 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:58.190195 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbdf5ad6-bc4c-40e6-8e84-74fd1b31b311 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:58.190858 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbdf5ad6-bc4c-40e6-8e84-74fd1b31b311 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:58.191132 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbdf5ad6-bc4c-40e6-8e84-74fd1b31b311 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:58.191398 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbdf5ad6-bc4c-40e6-8e84-74fd1b31b311 None admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:58.192037 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbdf5ad6-bc4c-40e6-8e84-74fd1b31b311 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:58.192328 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbdf5ad6-bc4c-40e6-8e84-74fd1b31b311 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:58.194287 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbdf5ad6-bc4c-40e6-8e84-74fd1b31b311 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:58.206918 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 331/653] 10.222.0.36 () {56 vars in 1109 bytes} [Thu Oct 20 08:48:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:58.212747 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aafa33b8-8c7a-4e02-ae0c-7788b2234c0a None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:58.266956 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:58.322073 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aafa33b8-8c7a-4e02-ae0c-7788b2234c0a None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:58.357080 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c83a594-0aa4-4356-a41c-0ac188c03338 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:58.357736 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c83a594-0aa4-4356-a41c-0ac188c03338 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:58.358010 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c83a594-0aa4-4356-a41c-0ac188c03338 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:58.358279 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c83a594-0aa4-4356-a41c-0ac188c03338 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:58.367276 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c83a594-0aa4-4356-a41c-0ac188c03338 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:58.374502 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c83a594-0aa4-4356-a41c-0ac188c03338 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:58.381886 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-9c83a594-0aa4-4356-a41c-0ac188c03338 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 20 08:48:58.381886 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:58.381886 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:58.381886 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:58.381886 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:58.381886 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:58.381886 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:58.381886 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:58.381886 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:58.381886 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:58.381886 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:58.381886 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:58.381886 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 20 08:48:58.381886 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 20 08:48:58.381886 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 20 08:48:58.381886 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 20 08:48:58.382886 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:58.382886 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:58.382886 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 20 08:48:58.382886 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 20 08:48:58.382886 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 20 08:48:58.382886 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 20 08:48:58.382886 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:58.382886 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:58.382886 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:58.382886 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:58.382886 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:58.382886 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:58.382886 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:58.382886 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:58.382886 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:58.382886 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:58.382886 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:58.382886 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:58.383859 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:58.383859 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:58.383859 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 20 08:48:58.383859 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 20 08:48:58.383859 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 20 08:48:58.383859 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 20 08:48:58.383859 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 20 08:48:58.383859 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 20 08:48:58.383859 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 20 08:48:58.383859 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:48:58.383859 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 323/654] 10.222.0.36 () {56 vars in 1117 bytes} [Thu Oct 20 08:48:58 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:58.389467 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbdf5ad6-bc4c-40e6-8e84-74fd1b31b311 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:58.462749 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:58.501207 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbdf5ad6-bc4c-40e6-8e84-74fd1b31b311 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:58.502763 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a624af58-adef-4ca1-841d-17005e631cdf None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:58.503049 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a624af58-adef-4ca1-841d-17005e631cdf None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:58.503322 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a624af58-adef-4ca1-841d-17005e631cdf None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:58.505446 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 324/655] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:48:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:48:58.512155 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c29a46-733c-4a56-9835-d5dffb9c5c5b None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:58.512438 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c29a46-733c-4a56-9835-d5dffb9c5c5b None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:58.512809 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c29a46-733c-4a56-9835-d5dffb9c5c5b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:58.532130 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-21c29a46-733c-4a56-9835-d5dffb9c5c5b None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:48:58.533456 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-21c29a46-733c-4a56-9835-d5dffb9c5c5b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:48:58.539617 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0869736-3946-4ed1-966d-6a2d4e618318 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:58.540211 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0869736-3946-4ed1-966d-6a2d4e618318 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:58.540483 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0869736-3946-4ed1-966d-6a2d4e618318 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:58.540747 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0869736-3946-4ed1-966d-6a2d4e618318 None admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:58.541457 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0869736-3946-4ed1-966d-6a2d4e618318 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:58.541752 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0869736-3946-4ed1-966d-6a2d4e618318 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:58.543779 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0869736-3946-4ed1-966d-6a2d4e618318 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:58.567935 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 332/656] 10.222.0.36 () {56 vars in 1149 bytes} [Thu Oct 20 08:48:58 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:58.573238 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0869736-3946-4ed1-966d-6a2d4e618318 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:58.618784 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 325/657] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:48:58 2022] POST /identity/v3/auth/tokens => generated 1271 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 20 08:48:58.679530 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0869736-3946-4ed1-966d-6a2d4e618318 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:58.705645 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:58.716492 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0430ea0-d556-4714-a041-f40b31e2d03f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:58.717111 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0430ea0-d556-4714-a041-f40b31e2d03f None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:58.717383 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0430ea0-d556-4714-a041-f40b31e2d03f None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:58.717651 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0430ea0-d556-4714-a041-f40b31e2d03f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:58.728825 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0430ea0-d556-4714-a041-f40b31e2d03f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:58.730263 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0430ea0-d556-4714-a041-f40b31e2d03f None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:58.745036 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-e0430ea0-d556-4714-a041-f40b31e2d03f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:58.745036 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:58.745036 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:58.745036 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:58.745036 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:58.745036 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:58.745036 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:58.745036 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:58.745036 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:58.745036 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:58.745036 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:58.745036 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:58.745036 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 20 08:48:58.745036 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 20 08:48:58.745036 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 20 08:48:58.745036 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 20 08:48:58.747332 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:58.747332 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:58.747332 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:58.747332 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:58.747332 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:58.747332 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:58.747332 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:58.747332 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:58.747332 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:58.747332 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:58.747332 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:58.747332 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:58.747332 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:58.747332 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:58.747332 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:58.747332 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:58.747332 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 20 08:48:58.749419 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 20 08:48:58.749419 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:48:58.749419 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:48:58.749419 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 333/658] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:48:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 20 08:48:58.759587 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c29a46-733c-4a56-9835-d5dffb9c5c5b None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:58.797293 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:58.867669 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c29a46-733c-4a56-9835-d5dffb9c5c5b None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:58.897434 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0430ea0-d556-4714-a041-f40b31e2d03f None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:58.905650 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68c59d28-58a2-4527-99e3-f9cf66aff777 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:58.906732 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c59d28-58a2-4527-99e3-f9cf66aff777 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:58.907202 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c59d28-58a2-4527-99e3-f9cf66aff777 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:58.907530 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c59d28-58a2-4527-99e3-f9cf66aff777 None admin] PATH_INFO: `/v3/domains` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:58.908554 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68c59d28-58a2-4527-99e3-f9cf66aff777 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:58.908857 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68c59d28-58a2-4527-99e3-f9cf66aff777 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:58.910232 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68c59d28-58a2-4527-99e3-f9cf66aff777 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:58.927388 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 326/659] 10.222.0.36 () {56 vars in 1109 bytes} [Thu Oct 20 08:48:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:58.928245 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68c59d28-58a2-4527-99e3-f9cf66aff777 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:59.017142 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0430ea0-d556-4714-a041-f40b31e2d03f None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:59.025950 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:59.032645 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68c59d28-58a2-4527-99e3-f9cf66aff777 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:59.061178 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b1d2b19-2199-4d07-b9fb-4259845e8a15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:59.062123 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b1d2b19-2199-4d07-b9fb-4259845e8a15 None admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:59.062659 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b1d2b19-2199-4d07-b9fb-4259845e8a15 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:59.063130 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b1d2b19-2199-4d07-b9fb-4259845e8a15 None admin] PATH_INFO: `/v3/services` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:59.064245 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b1d2b19-2199-4d07-b9fb-4259845e8a15 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:59.066370 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b1d2b19-2199-4d07-b9fb-4259845e8a15 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:59.069652 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29d806be-e8fd-41f2-ba3c-43eac74b1be7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:59.070594 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29d806be-e8fd-41f2-ba3c-43eac74b1be7 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:59.070997 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29d806be-e8fd-41f2-ba3c-43eac74b1be7 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:59.071271 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29d806be-e8fd-41f2-ba3c-43eac74b1be7 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:59.080739 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29d806be-e8fd-41f2-ba3c-43eac74b1be7 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:59.082269 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29d806be-e8fd-41f2-ba3c-43eac74b1be7 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:59.086800 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 334/660] 10.222.0.36 () {60 vars in 1141 bytes} [Thu Oct 20 08:48:58 2022] POST /identity/v3/services => generated 241 bytes in 190 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 20 08:48:59.108878 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-29d806be-e8fd-41f2-ba3c-43eac74b1be7 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 20 08:48:59.108878 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:48:59.108878 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:48:59.108878 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:48:59.108878 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:48:59.108878 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:48:59.108878 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:48:59.108878 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:48:59.108878 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:48:59.108878 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:48:59.108878 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:48:59.108878 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:48:59.108878 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 20 08:48:59.108878 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 20 08:48:59.108878 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 20 08:48:59.109865 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 20 08:48:59.109865 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:48:59.109865 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:48:59.109865 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:48:59.109865 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:48:59.109865 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:48:59.109865 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:48:59.109865 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:48:59.109865 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:48:59.109865 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:48:59.109865 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:48:59.109865 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:48:59.109865 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:48:59.109865 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:48:59.109865 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:48:59.109865 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:48:59.109865 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:48:59.110802 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 20 08:48:59.110802 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 20 08:48:59.110802 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 20 08:48:59.110802 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 20 08:48:59.110802 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 20 08:48:59.110802 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 20 08:48:59.110802 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 20 08:48:59.110802 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:48:59.110802 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 327/661] 10.222.0.36 () {56 vars in 1111 bytes} [Thu Oct 20 08:48:58 2022] GET /identity/v3/projects/service => generated 88 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:48:59.112979 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b1d2b19-2199-4d07-b9fb-4259845e8a15 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:59.163238 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:59.239291 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b1d2b19-2199-4d07-b9fb-4259845e8a15 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:59.278516 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a14e6d8-2653-4c8d-8e84-f6fdcaa547de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:59.278516 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a14e6d8-2653-4c8d-8e84-f6fdcaa547de None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:59.278842 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a14e6d8-2653-4c8d-8e84-f6fdcaa547de None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:59.279040 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a14e6d8-2653-4c8d-8e84-f6fdcaa547de None admin] PATH_INFO: `/v3/projects` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:59.279730 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a14e6d8-2653-4c8d-8e84-f6fdcaa547de None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:48:59.280026 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a14e6d8-2653-4c8d-8e84-f6fdcaa547de None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:59.281454 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a14e6d8-2653-4c8d-8e84-f6fdcaa547de None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:59.295285 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 335/662] 10.222.0.36 () {56 vars in 1148 bytes} [Thu Oct 20 08:48:59 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:48:59.301425 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29d806be-e8fd-41f2-ba3c-43eac74b1be7 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:48:59.418945 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29d806be-e8fd-41f2-ba3c-43eac74b1be7 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:48:59.455446 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-779f8158-2cf9-4df5-9468-cd83db66777f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:48:59.456455 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779f8158-2cf9-4df5-9468-cd83db66777f None admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:59.456966 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779f8158-2cf9-4df5-9468-cd83db66777f None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:59.457432 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779f8158-2cf9-4df5-9468-cd83db66777f None admin] PATH_INFO: `/v3/projects/d9857b610fb542969cfb5efbbdb8ce61/users/f4b0e9187ce74f7888e9e7094be5c824/roles/37a51151d3c042ffbaca4dc2bf317baf` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:59.460143 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-779f8158-2cf9-4df5-9468-cd83db66777f None admin] RBAC: Authorizing `identity:create_grant(project_id=d9857b610fb542969cfb5efbbdb8ce61, user_id=f4b0e9187ce74f7888e9e7094be5c824, role_id=37a51151d3c042ffbaca4dc2bf317baf)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:48:59.461844 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-779f8158-2cf9-4df5-9468-cd83db66777f None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:48:59.476351 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 328/663] 10.222.0.36 () {58 vars in 1439 bytes} [Thu Oct 20 08:48:59 2022] PUT /identity/v3/projects/d9857b610fb542969cfb5efbbdb8ce61/users/f4b0e9187ce74f7888e9e7094be5c824/roles/37a51151d3c042ffbaca4dc2bf317baf => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 08:48:59.701732 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bfd82b0-b42d-4f7f-9919-344955633e35 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:59.702080 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bfd82b0-b42d-4f7f-9919-344955633e35 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:59.702247 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bfd82b0-b42d-4f7f-9919-344955633e35 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:59.703788 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 336/664] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:48:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:48:59.710211 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae6073b-bb45-4558-a8f4-e8b857e1e1f3 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:59.710211 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae6073b-bb45-4558-a8f4-e8b857e1e1f3 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:59.710211 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae6073b-bb45-4558-a8f4-e8b857e1e1f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:59.729707 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-bae6073b-bb45-4558-a8f4-e8b857e1e1f3 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:48:59.731412 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-bae6073b-bb45-4558-a8f4-e8b857e1e1f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:48:59.743620 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:48:59.747753 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dd5e45f-7614-478a-bdae-54cebfaf5717 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:59.748042 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dd5e45f-7614-478a-bdae-54cebfaf5717 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:59.748308 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dd5e45f-7614-478a-bdae-54cebfaf5717 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:59.749922 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 337/665] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:48:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:48:59.755300 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c686904-fda4-4ab3-a755-9412bea884ed None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:48:59.755582 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c686904-fda4-4ab3-a755-9412bea884ed None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:48:59.755856 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c686904-fda4-4ab3-a755-9412bea884ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:48:59.774145 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-3c686904-fda4-4ab3-a755-9412bea884ed None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:48:59.775357 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-3c686904-fda4-4ab3-a755-9412bea884ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:48:59.820590 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 329/666] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:48:59 2022] POST /identity/v3/auth/tokens => generated 1373 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 20 08:48:59.850859 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 338/667] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:48:59 2022] POST /identity/v3/auth/tokens => generated 1373 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 20 08:49:00.084471 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bae6073b-bb45-4558-a8f4-e8b857e1e1f3 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:00.119189 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c686904-fda4-4ab3-a755-9412bea884ed None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:00.144764 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:00.210186 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bae6073b-bb45-4558-a8f4-e8b857e1e1f3 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:00.231044 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c686904-fda4-4ab3-a755-9412bea884ed None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:00.257527 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad1b8630-5511-4bf3-8d9b-617b2a62abc8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:00.258255 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad1b8630-5511-4bf3-8d9b-617b2a62abc8 None admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:00.258626 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad1b8630-5511-4bf3-8d9b-617b2a62abc8 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:00.259215 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad1b8630-5511-4bf3-8d9b-617b2a62abc8 None admin] PATH_INFO: `/v3/services` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:00.259949 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad1b8630-5511-4bf3-8d9b-617b2a62abc8 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:00.262120 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad1b8630-5511-4bf3-8d9b-617b2a62abc8 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:00.272060 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0acaf883-877b-415a-aec7-6baef6ead83f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:00.272683 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0acaf883-877b-415a-aec7-6baef6ead83f None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:00.273020 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0acaf883-877b-415a-aec7-6baef6ead83f None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:00.273251 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0acaf883-877b-415a-aec7-6baef6ead83f None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:00.273832 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0acaf883-877b-415a-aec7-6baef6ead83f None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:00.275228 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0acaf883-877b-415a-aec7-6baef6ead83f None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:00.286847 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-0acaf883-877b-415a-aec7-6baef6ead83f None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 20 08:49:00.286847 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:49:00.286847 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 20 08:49:00.286847 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 20 08:49:00.286847 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 20 08:49:00.286847 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 20 08:49:00.286847 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 20 08:49:00.286847 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 20 08:49:00.286847 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 20 08:49:00.286847 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:49:00.286847 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 20 08:49:00.286847 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:49:00.286847 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:49:00.286847 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:49:00.286847 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:49:00.286847 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:49:00.288093 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:49:00.288093 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:49:00.288093 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:49:00.288093 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:49:00.288093 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:49:00.288093 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:49:00.288093 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:49:00.288093 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 20 08:49:00.288093 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 20 08:49:00.288093 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 20 08:49:00.288093 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 20 08:49:00.288093 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:49:00.288093 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:49:00.288093 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:49:00.288093 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:49:00.288093 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:49:00.288093 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:49:00.289015 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:49:00.289015 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:49:00.289015 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:49:00.289015 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:49:00.289015 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:49:00.289015 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:49:00.289015 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:49:00.289015 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:49:00.289015 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:49:00.289015 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:49:00.289015 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 20 08:49:00.289015 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 20 08:49:00.289015 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 20 08:49:00.289015 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:49:00.290344 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 330/669] 10.222.0.36 () {60 vars in 1141 bytes} [Thu Oct 20 08:49:00 2022] POST /identity/v3/services => generated 251 bytes in 200 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 08:49:00.294515 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 339/669] 10.222.0.36 () {56 vars in 1132 bytes} [Thu Oct 20 08:49:00 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 20 08:49:00.295861 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad1b8630-5511-4bf3-8d9b-617b2a62abc8 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:00.317099 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:00.434266 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad1b8630-5511-4bf3-8d9b-617b2a62abc8 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:00.473171 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f47d9c-cfa7-4a29-9a07-9eb478135df8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:00.474158 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f47d9c-cfa7-4a29-9a07-9eb478135df8 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:00.474684 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f47d9c-cfa7-4a29-9a07-9eb478135df8 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:00.475154 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f47d9c-cfa7-4a29-9a07-9eb478135df8 None admin] PATH_INFO: `/v3/services` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:00.475994 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1f47d9c-cfa7-4a29-9a07-9eb478135df8 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:49:00.476486 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1f47d9c-cfa7-4a29-9a07-9eb478135df8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:00.478065 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1f47d9c-cfa7-4a29-9a07-9eb478135df8 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:00.484135 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 331/670] 10.222.0.36 () {56 vars in 1126 bytes} [Thu Oct 20 08:49:00 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:49:00.490311 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0acaf883-877b-415a-aec7-6baef6ead83f None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:00.587758 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:00.606507 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0acaf883-877b-415a-aec7-6baef6ead83f None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:00.641895 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25a85560-c53b-4561-ae5b-4d5366fb3fb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:00.642848 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a85560-c53b-4561-ae5b-4d5366fb3fb5 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:00.643357 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a85560-c53b-4561-ae5b-4d5366fb3fb5 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:00.643829 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a85560-c53b-4561-ae5b-4d5366fb3fb5 None admin] PATH_INFO: `/v3/services` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:00.644666 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25a85560-c53b-4561-ae5b-4d5366fb3fb5 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:49:00.645170 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25a85560-c53b-4561-ae5b-4d5366fb3fb5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:00.647321 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25a85560-c53b-4561-ae5b-4d5366fb3fb5 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:00.652958 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 340/671] 10.222.0.36 () {56 vars in 1126 bytes} [Thu Oct 20 08:49:00 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:49:00.658952 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f47d9c-cfa7-4a29-9a07-9eb478135df8 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:00.727950 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:00.765196 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f47d9c-cfa7-4a29-9a07-9eb478135df8 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:00.799890 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-926dab9c-20b2-4d31-b826-423546aade50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:00.800499 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926dab9c-20b2-4d31-b826-423546aade50 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:00.800777 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926dab9c-20b2-4d31-b826-423546aade50 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:00.801049 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926dab9c-20b2-4d31-b826-423546aade50 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:00.801751 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-926dab9c-20b2-4d31-b826-423546aade50 None admin] RBAC: Adding query filter params (interface=public, service_id=7ee16900dddd4e84b18e2692569b761d, region_id=RegionOne) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:49:00.802625 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-926dab9c-20b2-4d31-b826-423546aade50 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:00.804004 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-926dab9c-20b2-4d31-b826-423546aade50 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:00.809755 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 332/672] 10.222.0.36 () {56 vars in 1251 bytes} [Thu Oct 20 08:49:00 2022] GET /identity/v3/endpoints?service_id=7ee16900dddd4e84b18e2692569b761d&interface=public®ion_id=RegionOne => generated 194 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:49:00.815533 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25a85560-c53b-4561-ae5b-4d5366fb3fb5 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:00.920861 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25a85560-c53b-4561-ae5b-4d5366fb3fb5 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:00.961544 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5597fbe6-797d-44af-9f14-971d17e8c456 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:00.962147 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5597fbe6-797d-44af-9f14-971d17e8c456 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:00.962424 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5597fbe6-797d-44af-9f14-971d17e8c456 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:00.962726 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5597fbe6-797d-44af-9f14-971d17e8c456 None admin] PATH_INFO: `/v3/services` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:00.963325 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5597fbe6-797d-44af-9f14-971d17e8c456 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:00.964682 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5597fbe6-797d-44af-9f14-971d17e8c456 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:00.971131 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 341/673] 10.222.0.36 () {56 vars in 1087 bytes} [Thu Oct 20 08:49:00 2022] GET /identity/v3/services => generated 1497 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 08:49:01.421402 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b569a22-dbab-43d1-9724-680d0be81e21 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:01.422375 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b569a22-dbab-43d1-9724-680d0be81e21 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:01.422909 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b569a22-dbab-43d1-9724-680d0be81e21 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:01.424945 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 333/674] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:49:01 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:49:01.430505 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e6c99d-d6b4-4104-b6d6-1440a934d7c7 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:01.431339 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e6c99d-d6b4-4104-b6d6-1440a934d7c7 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:01.431817 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e6c99d-d6b4-4104-b6d6-1440a934d7c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:01.451550 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-e6e6c99d-d6b4-4104-b6d6-1440a934d7c7 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:49:01.453050 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-e6e6c99d-d6b4-4104-b6d6-1440a934d7c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:49:01.465934 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:01.527649 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 342/675] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:49:01 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 20 08:49:01.654966 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98b9d539-a25f-4cb2-ad43-f54e60732351 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:01.655210 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98b9d539-a25f-4cb2-ad43-f54e60732351 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:01.655443 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98b9d539-a25f-4cb2-ad43-f54e60732351 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:01.657030 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 334/676] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:49:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:49:01.662752 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fabe5e3f-716e-41cd-a9f9-8d9ba4f8a26a None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:01.663041 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fabe5e3f-716e-41cd-a9f9-8d9ba4f8a26a None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:01.663314 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fabe5e3f-716e-41cd-a9f9-8d9ba4f8a26a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:01.682856 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-fabe5e3f-716e-41cd-a9f9-8d9ba4f8a26a None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:49:01.684080 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-fabe5e3f-716e-41cd-a9f9-8d9ba4f8a26a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:49:01.757362 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 343/677] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:49:01 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 20 08:49:01.813234 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98b9d539-a25f-4cb2-ad43-f54e60732351 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:01.923820 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98b9d539-a25f-4cb2-ad43-f54e60732351 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:01.954218 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:01.960679 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04b3d9eb-5082-439d-b61f-c7fc4e4915ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:01.961491 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b3d9eb-5082-439d-b61f-c7fc4e4915ba None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:01.962000 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b3d9eb-5082-439d-b61f-c7fc4e4915ba None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:01.962493 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b3d9eb-5082-439d-b61f-c7fc4e4915ba None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:01.963081 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04b3d9eb-5082-439d-b61f-c7fc4e4915ba None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:01.964523 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04b3d9eb-5082-439d-b61f-c7fc4e4915ba None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:01.973578 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-04b3d9eb-5082-439d-b61f-c7fc4e4915ba None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 20 08:49:01.973578 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:49:01.973578 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 20 08:49:01.973578 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 20 08:49:01.973578 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 20 08:49:01.973578 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 20 08:49:01.973578 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 20 08:49:01.973578 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 20 08:49:01.973578 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 20 08:49:01.973578 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:49:01.973578 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 20 08:49:01.973578 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:49:01.973578 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:49:01.973578 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:49:01.973578 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:49:01.973578 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:49:01.973578 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:49:01.975106 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:49:01.975106 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:49:01.975106 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:49:01.975106 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:49:01.975106 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:49:01.975106 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:49:01.975106 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 20 08:49:01.975106 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 20 08:49:01.975106 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 20 08:49:01.975106 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 20 08:49:01.975106 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:49:01.975106 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:49:01.975106 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:49:01.975106 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:49:01.975106 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:49:01.975106 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:49:01.975967 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:49:01.975967 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:49:01.975967 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:49:01.975967 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:49:01.975967 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:49:01.975967 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:49:01.975967 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:49:01.975967 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:49:01.975967 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:49:01.975967 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:49:01.975967 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 20 08:49:01.975967 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 20 08:49:01.975967 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 20 08:49:01.975967 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:49:01.975967 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 335/678] 10.222.0.36 () {56 vars in 1126 bytes} [Thu Oct 20 08:49:01 2022] GET /identity/v3/services/object-store => generated 93 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:49:01.981506 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fabe5e3f-716e-41cd-a9f9-8d9ba4f8a26a None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:02.037227 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04b3d9eb-5082-439d-b61f-c7fc4e4915ba None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:02.063273 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:02.090489 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fabe5e3f-716e-41cd-a9f9-8d9ba4f8a26a None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:02.136478 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-395d8586-3d46-44ef-b84d-237d9b8614ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:02.137090 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-395d8586-3d46-44ef-b84d-237d9b8614ad None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:02.137468 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-395d8586-3d46-44ef-b84d-237d9b8614ad None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:02.137927 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-395d8586-3d46-44ef-b84d-237d9b8614ad None admin] PATH_INFO: `/v3/services` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:02.138702 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-395d8586-3d46-44ef-b84d-237d9b8614ad None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:49:02.139011 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-395d8586-3d46-44ef-b84d-237d9b8614ad None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:02.140386 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-395d8586-3d46-44ef-b84d-237d9b8614ad None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:02.148200 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04b3d9eb-5082-439d-b61f-c7fc4e4915ba None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:02.150546 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 344/679] 10.222.0.36 () {56 vars in 1122 bytes} [Thu Oct 20 08:49:01 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:49:02.152572 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-395d8586-3d46-44ef-b84d-237d9b8614ad None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:02.200232 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61079fa9-2cfb-4ae3-82fe-3d13e8299ffa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:02.201544 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61079fa9-2cfb-4ae3-82fe-3d13e8299ffa None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:02.202063 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61079fa9-2cfb-4ae3-82fe-3d13e8299ffa None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:02.202913 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61079fa9-2cfb-4ae3-82fe-3d13e8299ffa None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:02.218845 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61079fa9-2cfb-4ae3-82fe-3d13e8299ffa None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 20 08:49:02.227313 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-61079fa9-2cfb-4ae3-82fe-3d13e8299ffa None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 20 08:49:02.227313 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:49:02.227313 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:49:02.227313 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:49:02.227313 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:49:02.227313 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:49:02.227313 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:49:02.227313 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:49:02.227313 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:49:02.227313 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:49:02.227313 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:49:02.227313 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:49:02.227313 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 20 08:49:02.227313 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 20 08:49:02.227313 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 20 08:49:02.227313 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 20 08:49:02.228324 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 20 08:49:02.228324 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 20 08:49:02.228324 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 20 08:49:02.228324 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 20 08:49:02.228324 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:49:02.228324 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:49:02.228324 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:49:02.228324 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:49:02.228324 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:49:02.228324 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:49:02.228324 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:49:02.228324 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:49:02.228324 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:49:02.228324 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:49:02.228324 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:49:02.228324 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:49:02.229237 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:49:02.229237 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:49:02.229237 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:49:02.229237 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:49:02.229237 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 20 08:49:02.229237 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 20 08:49:02.229237 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 20 08:49:02.229237 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 20 08:49:02.229237 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 20 08:49:02.229237 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 20 08:49:02.229237 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 20 08:49:02.229237 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:49:02.232810 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 336/680] 10.222.0.36 () {56 vars in 1120 bytes} [Thu Oct 20 08:49:02 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 20 08:49:02.238556 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61079fa9-2cfb-4ae3-82fe-3d13e8299ffa None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:02.268675 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-395d8586-3d46-44ef-b84d-237d9b8614ad None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:02.306281 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04fe8c49-72a3-4a5d-a5b5-ffb45fe61109 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:02.307179 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04fe8c49-72a3-4a5d-a5b5-ffb45fe61109 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:02.307294 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04fe8c49-72a3-4a5d-a5b5-ffb45fe61109 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:02.307480 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04fe8c49-72a3-4a5d-a5b5-ffb45fe61109 None admin] PATH_INFO: `/v3/services` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:02.308115 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04fe8c49-72a3-4a5d-a5b5-ffb45fe61109 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:49:02.308415 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04fe8c49-72a3-4a5d-a5b5-ffb45fe61109 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:02.309853 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04fe8c49-72a3-4a5d-a5b5-ffb45fe61109 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:02.320108 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 345/681] 10.222.0.36 () {56 vars in 1122 bytes} [Thu Oct 20 08:49:02 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 08:49:02.321430 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04fe8c49-72a3-4a5d-a5b5-ffb45fe61109 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:02.324719 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:02.326606 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:02.398131 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61079fa9-2cfb-4ae3-82fe-3d13e8299ffa None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:02.437206 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04fe8c49-72a3-4a5d-a5b5-ffb45fe61109 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:02.447402 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14012212-60e3-4e84-b9cc-a9370415ad9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:02.448422 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14012212-60e3-4e84-b9cc-a9370415ad9d None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:02.448980 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14012212-60e3-4e84-b9cc-a9370415ad9d None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:02.449262 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14012212-60e3-4e84-b9cc-a9370415ad9d None admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:02.449932 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14012212-60e3-4e84-b9cc-a9370415ad9d None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:49:02.450311 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14012212-60e3-4e84-b9cc-a9370415ad9d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:02.452492 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14012212-60e3-4e84-b9cc-a9370415ad9d None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:02.466608 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 337/682] 10.222.0.36 () {56 vars in 1115 bytes} [Thu Oct 20 08:49:02 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:49:02.470094 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14012212-60e3-4e84-b9cc-a9370415ad9d None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:02.479603 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02d0ac62-5d4b-4091-8dbf-fe44a2beee36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:02.480199 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02d0ac62-5d4b-4091-8dbf-fe44a2beee36 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:02.480471 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02d0ac62-5d4b-4091-8dbf-fe44a2beee36 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:02.480735 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02d0ac62-5d4b-4091-8dbf-fe44a2beee36 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:02.481475 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02d0ac62-5d4b-4091-8dbf-fe44a2beee36 None admin] RBAC: Adding query filter params (interface=public, service_id=23da7943ed384de8bd44e33228d6ccc2, region_id=RegionOne) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:49:02.481772 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02d0ac62-5d4b-4091-8dbf-fe44a2beee36 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:02.483496 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02d0ac62-5d4b-4091-8dbf-fe44a2beee36 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:02.488788 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 346/683] 10.222.0.36 () {56 vars in 1251 bytes} [Thu Oct 20 08:49:02 2022] GET /identity/v3/endpoints?service_id=23da7943ed384de8bd44e33228d6ccc2&interface=public®ion_id=RegionOne => generated 194 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 08:49:02.494240 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02d0ac62-5d4b-4091-8dbf-fe44a2beee36 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:02.582541 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:02.589167 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14012212-60e3-4e84-b9cc-a9370415ad9d None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:02.609074 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:02.628368 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02d0ac62-5d4b-4091-8dbf-fe44a2beee36 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:02.629541 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcc39a73-a65d-4974-9275-f7fdd74d2f03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:02.630392 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcc39a73-a65d-4974-9275-f7fdd74d2f03 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:02.630786 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcc39a73-a65d-4974-9275-f7fdd74d2f03 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:02.631149 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcc39a73-a65d-4974-9275-f7fdd74d2f03 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:02.643648 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcc39a73-a65d-4974-9275-f7fdd74d2f03 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:02.645346 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcc39a73-a65d-4974-9275-f7fdd74d2f03 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:02.662218 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-dcc39a73-a65d-4974-9275-f7fdd74d2f03 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:49:02.662218 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:49:02.662218 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:49:02.662218 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:49:02.662218 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:49:02.662218 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:49:02.662218 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:49:02.662218 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:49:02.662218 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:49:02.662218 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:49:02.662218 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:49:02.662218 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:49:02.662218 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 20 08:49:02.662218 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 20 08:49:02.662218 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 20 08:49:02.662218 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 20 08:49:02.663330 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:49:02.663330 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:49:02.663330 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:49:02.663330 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:49:02.663330 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:49:02.663330 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:49:02.663330 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:49:02.663330 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:49:02.663330 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:49:02.663330 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:49:02.663330 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:49:02.663330 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:49:02.663330 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:49:02.663330 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:49:02.663330 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:49:02.663330 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:49:02.663330 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 20 08:49:02.664265 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 20 08:49:02.664265 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:49:02.664265 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:49:02.668712 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 338/684] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:49:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:49:02.669726 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcc39a73-a65d-4974-9275-f7fdd74d2f03 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:02.671418 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-146abbd4-c932-4ef6-b044-9903b8c42c61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:02.672262 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-146abbd4-c932-4ef6-b044-9903b8c42c61 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:02.672823 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-146abbd4-c932-4ef6-b044-9903b8c42c61 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:02.673271 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-146abbd4-c932-4ef6-b044-9903b8c42c61 None admin] PATH_INFO: `/v3/services` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:02.674165 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-146abbd4-c932-4ef6-b044-9903b8c42c61 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:02.675762 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-146abbd4-c932-4ef6-b044-9903b8c42c61 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:02.682867 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 347/685] 10.222.0.36 () {56 vars in 1087 bytes} [Thu Oct 20 08:49:02 2022] GET /identity/v3/services => generated 1497 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 08:49:02.684394 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733701fc-b54d-4269-84d9-664b7323627f None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:02.684872 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733701fc-b54d-4269-84d9-664b7323627f None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:02.685213 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733701fc-b54d-4269-84d9-664b7323627f None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:02.686675 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 348/686] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:49:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:49:02.693704 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-872bbe89-afbb-4322-9e05-5b664a1e7b58 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:02.694069 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-872bbe89-afbb-4322-9e05-5b664a1e7b58 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:02.694439 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-872bbe89-afbb-4322-9e05-5b664a1e7b58 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:02.718007 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-872bbe89-afbb-4322-9e05-5b664a1e7b58 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:49:02.719201 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-872bbe89-afbb-4322-9e05-5b664a1e7b58 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:49:02.806988 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcc39a73-a65d-4974-9275-f7fdd74d2f03 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:02.810768 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 349/687] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:49:02 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 20 08:49:02.846373 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0868c4f1-29d3-4cce-a48d-2fa4f391dcc9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:02.847382 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0868c4f1-29d3-4cce-a48d-2fa4f391dcc9 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:02.847859 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0868c4f1-29d3-4cce-a48d-2fa4f391dcc9 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:02.848348 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0868c4f1-29d3-4cce-a48d-2fa4f391dcc9 None admin] PATH_INFO: `/v3/domains` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:02.849246 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0868c4f1-29d3-4cce-a48d-2fa4f391dcc9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:49:02.849746 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0868c4f1-29d3-4cce-a48d-2fa4f391dcc9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:02.851375 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0868c4f1-29d3-4cce-a48d-2fa4f391dcc9 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:02.865145 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 339/688] 10.222.0.36 () {56 vars in 1109 bytes} [Thu Oct 20 08:49:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:49:02.871628 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-872bbe89-afbb-4322-9e05-5b664a1e7b58 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:02.935316 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:02.975781 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-872bbe89-afbb-4322-9e05-5b664a1e7b58 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:03.010124 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d6e412e-7bc8-46e2-97d8-9e82a2658026 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:03.010782 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d6e412e-7bc8-46e2-97d8-9e82a2658026 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:03.011055 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d6e412e-7bc8-46e2-97d8-9e82a2658026 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:03.011318 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d6e412e-7bc8-46e2-97d8-9e82a2658026 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:03.018635 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d6e412e-7bc8-46e2-97d8-9e82a2658026 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:03.020070 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d6e412e-7bc8-46e2-97d8-9e82a2658026 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:03.032718 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-5d6e412e-7bc8-46e2-97d8-9e82a2658026 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 20 08:49:03.032718 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:49:03.032718 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:49:03.032718 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:49:03.032718 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:49:03.032718 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:49:03.032718 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:49:03.032718 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:49:03.032718 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:49:03.032718 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:49:03.032718 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:49:03.032718 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:49:03.032718 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 20 08:49:03.032718 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 20 08:49:03.032718 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 20 08:49:03.032718 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 20 08:49:03.033696 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:49:03.033696 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:49:03.033696 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 20 08:49:03.033696 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 20 08:49:03.033696 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 20 08:49:03.033696 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 20 08:49:03.033696 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:49:03.033696 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:49:03.033696 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:49:03.033696 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:49:03.033696 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:49:03.033696 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:49:03.033696 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:49:03.033696 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:49:03.033696 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:49:03.033696 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:49:03.033696 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:49:03.033696 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:49:03.035352 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:49:03.035352 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:49:03.035352 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 20 08:49:03.035352 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 20 08:49:03.035352 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 20 08:49:03.035352 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 20 08:49:03.035352 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 20 08:49:03.035352 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 20 08:49:03.035352 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 20 08:49:03.035352 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:49:03.035352 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 350/689] 10.222.0.36 () {56 vars in 1117 bytes} [Thu Oct 20 08:49:02 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:49:03.039713 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0868c4f1-29d3-4cce-a48d-2fa4f391dcc9 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:03.071173 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:03.092339 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d6e412e-7bc8-46e2-97d8-9e82a2658026 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:03.160046 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0868c4f1-29d3-4cce-a48d-2fa4f391dcc9 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:03.205697 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82179f0e-2dff-4e9b-a890-88d951742ea9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:03.206064 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d6e412e-7bc8-46e2-97d8-9e82a2658026 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:03.206980 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82179f0e-2dff-4e9b-a890-88d951742ea9 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:03.207467 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82179f0e-2dff-4e9b-a890-88d951742ea9 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:03.208648 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82179f0e-2dff-4e9b-a890-88d951742ea9 None admin] PATH_INFO: `/v3/users` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:03.210526 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82179f0e-2dff-4e9b-a890-88d951742ea9 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:49:03.210844 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82179f0e-2dff-4e9b-a890-88d951742ea9 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:03.212282 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82179f0e-2dff-4e9b-a890-88d951742ea9 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:03.232101 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 340/690] 10.222.0.36 () {56 vars in 1149 bytes} [Thu Oct 20 08:49:03 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:49:03.232263 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:03.233252 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6733b7e-5834-4a4e-811c-528d351e457b None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:03.233522 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6733b7e-5834-4a4e-811c-528d351e457b None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:03.233806 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6733b7e-5834-4a4e-811c-528d351e457b None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:03.238656 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 341/691] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:49:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:49:03.240205 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6733b7e-5834-4a4e-811c-528d351e457b None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:03.248717 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19e2f2e9-65ed-4755-ae7c-1165f9f400f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:03.249569 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e2f2e9-65ed-4755-ae7c-1165f9f400f8 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:03.249851 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e2f2e9-65ed-4755-ae7c-1165f9f400f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:03.250118 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e2f2e9-65ed-4755-ae7c-1165f9f400f8 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:03.250745 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19e2f2e9-65ed-4755-ae7c-1165f9f400f8 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:03.252117 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19e2f2e9-65ed-4755-ae7c-1165f9f400f8 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:03.260989 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-19e2f2e9-65ed-4755-ae7c-1165f9f400f8 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 20 08:49:03.260989 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:49:03.260989 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 20 08:49:03.260989 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 20 08:49:03.260989 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 20 08:49:03.260989 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 20 08:49:03.260989 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 20 08:49:03.260989 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 20 08:49:03.260989 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 20 08:49:03.260989 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:49:03.260989 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 20 08:49:03.260989 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:49:03.260989 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:49:03.260989 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:49:03.260989 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:49:03.260989 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:49:03.260989 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:49:03.262026 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:49:03.262026 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:49:03.262026 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:49:03.262026 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:49:03.262026 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:49:03.262026 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:49:03.262026 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 20 08:49:03.262026 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 20 08:49:03.262026 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 20 08:49:03.262026 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 20 08:49:03.262026 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:49:03.262026 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:49:03.262026 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:49:03.262026 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:49:03.262026 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:49:03.262026 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:49:03.262932 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:49:03.262932 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:49:03.262932 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:49:03.262932 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:49:03.262932 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:49:03.262932 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:49:03.262932 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:49:03.262932 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:49:03.262932 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:49:03.262932 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:49:03.262932 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 20 08:49:03.262932 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 20 08:49:03.262932 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 20 08:49:03.262932 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:49:03.262932 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 351/692] 10.222.0.36 () {56 vars in 1114 bytes} [Thu Oct 20 08:49:03 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:49:03.264202 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a0ad96-d855-4253-bd89-1db0f0a968c9 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:03.264202 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a0ad96-d855-4253-bd89-1db0f0a968c9 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:03.264202 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a0ad96-d855-4253-bd89-1db0f0a968c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:03.282379 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-b6a0ad96-d855-4253-bd89-1db0f0a968c9 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:49:03.283616 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-b6a0ad96-d855-4253-bd89-1db0f0a968c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:49:03.337378 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:03.355523 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6733b7e-5834-4a4e-811c-528d351e457b None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:03.356539 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 352/693] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:49:03 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 20 08:49:03.359362 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a0ad96-d855-4253-bd89-1db0f0a968c9 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:03.404226 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db89d05d-d29e-4bb1-8649-4e6ec0d4e7b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:03.404838 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db89d05d-d29e-4bb1-8649-4e6ec0d4e7b9 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:03.405118 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db89d05d-d29e-4bb1-8649-4e6ec0d4e7b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:03.405387 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db89d05d-d29e-4bb1-8649-4e6ec0d4e7b9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:03.485104 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db89d05d-d29e-4bb1-8649-4e6ec0d4e7b9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:03.486499 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db89d05d-d29e-4bb1-8649-4e6ec0d4e7b9 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:03.495541 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a0ad96-d855-4253-bd89-1db0f0a968c9 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:03.512405 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:03.532936 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5018a51c-3e7f-471d-9d16-af588dc44052 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:03.533544 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5018a51c-3e7f-471d-9d16-af588dc44052 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:03.533823 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5018a51c-3e7f-471d-9d16-af588dc44052 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:03.534097 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5018a51c-3e7f-471d-9d16-af588dc44052 None admin] PATH_INFO: `/v3/services` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:03.534767 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5018a51c-3e7f-471d-9d16-af588dc44052 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:49:03.535114 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5018a51c-3e7f-471d-9d16-af588dc44052 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:03.537072 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5018a51c-3e7f-471d-9d16-af588dc44052 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:03.542288 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 353/694] 10.222.0.36 () {56 vars in 1114 bytes} [Thu Oct 20 08:49:03 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:49:03.547668 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5018a51c-3e7f-471d-9d16-af588dc44052 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:03.597362 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:03.620038 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 342/695] 10.222.0.36 () {58 vars in 1282 bytes} [Thu Oct 20 08:49:03 2022] GET /identity/v3/auth/tokens => generated 1474 bytes in 380 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 20 08:49:03.625940 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db89d05d-d29e-4bb1-8649-4e6ec0d4e7b9 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:03.660810 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5018a51c-3e7f-471d-9d16-af588dc44052 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:03.694911 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69582cc9-f334-4b36-8aef-8f1795742e3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:03.695508 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69582cc9-f334-4b36-8aef-8f1795742e3d None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:03.695781 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69582cc9-f334-4b36-8aef-8f1795742e3d None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:03.696046 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69582cc9-f334-4b36-8aef-8f1795742e3d None admin] PATH_INFO: `/v3/services` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:03.696662 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69582cc9-f334-4b36-8aef-8f1795742e3d None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:49:03.696958 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69582cc9-f334-4b36-8aef-8f1795742e3d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:03.698312 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69582cc9-f334-4b36-8aef-8f1795742e3d None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:03.708375 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 354/696] 10.222.0.36 () {56 vars in 1114 bytes} [Thu Oct 20 08:49:03 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 08:49:03.708375 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69582cc9-f334-4b36-8aef-8f1795742e3d None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:03.735661 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db89d05d-d29e-4bb1-8649-4e6ec0d4e7b9 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:03.753996 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:03.772657 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e2dbd88-3786-4eeb-ac50-6d093d3b70f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:03.773269 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2dbd88-3786-4eeb-ac50-6d093d3b70f2 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:03.773545 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2dbd88-3786-4eeb-ac50-6d093d3b70f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:03.773815 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2dbd88-3786-4eeb-ac50-6d093d3b70f2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:03.783765 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e2dbd88-3786-4eeb-ac50-6d093d3b70f2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:03.785222 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e2dbd88-3786-4eeb-ac50-6d093d3b70f2 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:03.800264 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-2e2dbd88-3786-4eeb-ac50-6d093d3b70f2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:49:03.800264 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:49:03.800264 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:49:03.800264 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:49:03.800264 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:49:03.800264 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:49:03.800264 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:49:03.800264 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:49:03.800264 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:49:03.800264 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:49:03.800264 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:49:03.800264 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:49:03.800264 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 20 08:49:03.800264 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 20 08:49:03.800264 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 20 08:49:03.800264 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 20 08:49:03.801377 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:49:03.801377 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:49:03.801377 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:49:03.801377 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:49:03.801377 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:49:03.801377 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:49:03.801377 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:49:03.801377 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:49:03.801377 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:49:03.801377 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:49:03.801377 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:49:03.801377 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:49:03.801377 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:49:03.801377 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:49:03.801377 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:49:03.801377 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:49:03.801377 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 20 08:49:03.802913 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 20 08:49:03.802913 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:49:03.802913 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:49:03.802913 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 343/697] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:49:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:49:03.803489 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e2dbd88-3786-4eeb-ac50-6d093d3b70f2 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:03.814832 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69582cc9-f334-4b36-8aef-8f1795742e3d None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:03.843684 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:03.852998 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-433db967-b341-4191-8162-e0927b5a2e98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:03.853550 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-433db967-b341-4191-8162-e0927b5a2e98 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:03.853825 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-433db967-b341-4191-8162-e0927b5a2e98 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:03.854091 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-433db967-b341-4191-8162-e0927b5a2e98 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:03.854851 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-433db967-b341-4191-8162-e0927b5a2e98 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:03.856517 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-433db967-b341-4191-8162-e0927b5a2e98 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:03.866425 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-433db967-b341-4191-8162-e0927b5a2e98 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 20 08:49:03.866425 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:49:03.866425 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 20 08:49:03.866425 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 20 08:49:03.866425 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 20 08:49:03.866425 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 20 08:49:03.866425 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 20 08:49:03.866425 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 20 08:49:03.866425 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 20 08:49:03.866425 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:49:03.866425 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 20 08:49:03.866425 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:49:03.866425 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:49:03.866425 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:49:03.866425 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:49:03.866425 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:49:03.867397 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:49:03.867397 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:49:03.867397 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:49:03.867397 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:49:03.867397 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:49:03.867397 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:49:03.867397 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:49:03.867397 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 20 08:49:03.867397 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 20 08:49:03.867397 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 20 08:49:03.867397 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 20 08:49:03.867397 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:49:03.867397 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:49:03.867397 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:49:03.867397 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:49:03.867397 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:49:03.867397 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:49:03.868317 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:49:03.868317 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:49:03.868317 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:49:03.868317 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:49:03.868317 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:49:03.868317 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:49:03.868317 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:49:03.868317 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:49:03.868317 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:49:03.868317 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:49:03.868317 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 20 08:49:03.868317 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 20 08:49:03.868317 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 20 08:49:03.868317 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:49:03.868317 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 355/698] 10.222.0.36 () {56 vars in 1132 bytes} [Thu Oct 20 08:49:03 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:49:03.869131 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-433db967-b341-4191-8162-e0927b5a2e98 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:03.911588 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e2dbd88-3786-4eeb-ac50-6d093d3b70f2 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:03.948364 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-604eb95b-d590-4d27-8da3-8b485587fdfe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:03.948971 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-604eb95b-d590-4d27-8da3-8b485587fdfe None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:03.949247 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-604eb95b-d590-4d27-8da3-8b485587fdfe None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:03.949515 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-604eb95b-d590-4d27-8da3-8b485587fdfe None admin] PATH_INFO: `/v3/endpoints` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:03.950219 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-604eb95b-d590-4d27-8da3-8b485587fdfe None admin] RBAC: Adding query filter params (interface=public, service_id=f0526ff041de4115935228408b99b744, region_id=RegionOne) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:49:03.954897 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-604eb95b-d590-4d27-8da3-8b485587fdfe None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:03.954897 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-604eb95b-d590-4d27-8da3-8b485587fdfe None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:03.957088 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 344/699] 10.222.0.36 () {56 vars in 1251 bytes} [Thu Oct 20 08:49:03 2022] GET /identity/v3/endpoints?service_id=f0526ff041de4115935228408b99b744&interface=public®ion_id=RegionOne => generated 194 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:49:03.957773 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-604eb95b-d590-4d27-8da3-8b485587fdfe None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:03.981674 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-433db967-b341-4191-8162-e0927b5a2e98 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:03.993226 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:04.021244 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-409da6d3-5d38-4629-9e09-cf75ae2e3252 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:04.021866 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-409da6d3-5d38-4629-9e09-cf75ae2e3252 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:04.022140 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-409da6d3-5d38-4629-9e09-cf75ae2e3252 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:04.022410 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-409da6d3-5d38-4629-9e09-cf75ae2e3252 None admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:04.023071 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-409da6d3-5d38-4629-9e09-cf75ae2e3252 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:49:04.023368 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-409da6d3-5d38-4629-9e09-cf75ae2e3252 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:04.024725 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-409da6d3-5d38-4629-9e09-cf75ae2e3252 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:04.037346 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 356/700] 10.222.0.36 () {56 vars in 1109 bytes} [Thu Oct 20 08:49:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:49:04.038570 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-409da6d3-5d38-4629-9e09-cf75ae2e3252 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:04.063660 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-604eb95b-d590-4d27-8da3-8b485587fdfe None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:04.065167 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:04.105893 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddea92b9-8b2e-490d-8647-2847cb69e321 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:04.106749 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddea92b9-8b2e-490d-8647-2847cb69e321 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:04.107191 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddea92b9-8b2e-490d-8647-2847cb69e321 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:04.107464 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddea92b9-8b2e-490d-8647-2847cb69e321 None admin] PATH_INFO: `/v3/services` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:04.108092 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddea92b9-8b2e-490d-8647-2847cb69e321 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:49:04.108393 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddea92b9-8b2e-490d-8647-2847cb69e321 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:04.109773 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddea92b9-8b2e-490d-8647-2847cb69e321 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:04.115221 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 345/701] 10.222.0.36 () {56 vars in 1126 bytes} [Thu Oct 20 08:49:03 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:49:04.115994 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddea92b9-8b2e-490d-8647-2847cb69e321 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:04.151335 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-409da6d3-5d38-4629-9e09-cf75ae2e3252 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:04.188592 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f35d8436-9a81-4009-8ef7-a97abeef87dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:04.189183 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f35d8436-9a81-4009-8ef7-a97abeef87dd None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:04.189455 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f35d8436-9a81-4009-8ef7-a97abeef87dd None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:04.189721 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f35d8436-9a81-4009-8ef7-a97abeef87dd None admin] PATH_INFO: `/v3/services` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:04.190315 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f35d8436-9a81-4009-8ef7-a97abeef87dd None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:04.191682 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f35d8436-9a81-4009-8ef7-a97abeef87dd None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:04.206501 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 357/702] 10.222.0.36 () {56 vars in 1087 bytes} [Thu Oct 20 08:49:04 2022] GET /identity/v3/services => generated 1497 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 08:49:04.206501 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f35d8436-9a81-4009-8ef7-a97abeef87dd None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:04.229954 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddea92b9-8b2e-490d-8647-2847cb69e321 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:04.244668 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:04.275270 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28147afd-2186-4906-a9ad-7891fbed3eb0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:04.275931 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28147afd-2186-4906-a9ad-7891fbed3eb0 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:04.276206 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28147afd-2186-4906-a9ad-7891fbed3eb0 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:04.276770 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28147afd-2186-4906-a9ad-7891fbed3eb0 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:04.288315 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28147afd-2186-4906-a9ad-7891fbed3eb0 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:04.290109 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28147afd-2186-4906-a9ad-7891fbed3eb0 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:04.307712 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-28147afd-2186-4906-a9ad-7891fbed3eb0 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 20 08:49:04.307712 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:49:04.307712 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:49:04.307712 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:49:04.307712 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:49:04.307712 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:49:04.307712 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:49:04.307712 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:49:04.307712 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:49:04.307712 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:49:04.307712 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:49:04.307712 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:49:04.307712 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 20 08:49:04.307712 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 20 08:49:04.307712 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 20 08:49:04.308677 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 20 08:49:04.308677 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:49:04.308677 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:49:04.308677 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:49:04.308677 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:49:04.308677 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:49:04.308677 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:49:04.308677 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:49:04.308677 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:49:04.308677 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:49:04.308677 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:49:04.308677 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:49:04.308677 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:49:04.308677 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:49:04.308677 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:49:04.308677 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:49:04.308677 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:49:04.309578 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 20 08:49:04.309578 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 20 08:49:04.309578 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 20 08:49:04.309578 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 20 08:49:04.309578 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 20 08:49:04.309578 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 20 08:49:04.309578 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 20 08:49:04.309578 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:49:04.313143 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 346/703] 10.222.0.36 () {56 vars in 1111 bytes} [Thu Oct 20 08:49:04 2022] GET /identity/v3/projects/service => generated 88 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:49:04.316007 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28147afd-2186-4906-a9ad-7891fbed3eb0 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:04.338075 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f35d8436-9a81-4009-8ef7-a97abeef87dd None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:04.356966 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:04.389777 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87be4518-0113-45b2-bfa3-f628ec5558e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:04.390452 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87be4518-0113-45b2-bfa3-f628ec5558e6 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:04.390757 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87be4518-0113-45b2-bfa3-f628ec5558e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:04.391031 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87be4518-0113-45b2-bfa3-f628ec5558e6 None admin] PATH_INFO: `/v3/services` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:04.391669 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87be4518-0113-45b2-bfa3-f628ec5558e6 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:49:04.391972 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87be4518-0113-45b2-bfa3-f628ec5558e6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:04.393340 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87be4518-0113-45b2-bfa3-f628ec5558e6 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:04.400069 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 358/704] 10.222.0.36 () {56 vars in 1126 bytes} [Thu Oct 20 08:49:04 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:49:04.405215 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87be4518-0113-45b2-bfa3-f628ec5558e6 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:04.448613 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28147afd-2186-4906-a9ad-7891fbed3eb0 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:04.493322 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12c2841a-68d8-4084-afeb-678bd0ea13b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:04.494816 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12c2841a-68d8-4084-afeb-678bd0ea13b8 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:04.495386 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12c2841a-68d8-4084-afeb-678bd0ea13b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:04.495945 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12c2841a-68d8-4084-afeb-678bd0ea13b8 None admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:04.496852 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12c2841a-68d8-4084-afeb-678bd0ea13b8 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:49:04.497349 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12c2841a-68d8-4084-afeb-678bd0ea13b8 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:04.499032 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12c2841a-68d8-4084-afeb-678bd0ea13b8 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:04.518456 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 347/705] 10.222.0.36 () {56 vars in 1148 bytes} [Thu Oct 20 08:49:04 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:49:04.520121 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12c2841a-68d8-4084-afeb-678bd0ea13b8 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:04.523493 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:04.529457 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87be4518-0113-45b2-bfa3-f628ec5558e6 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:04.570456 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f4f1670-2676-45a9-b3d1-c872cf2b7c55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:04.571089 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4f1670-2676-45a9-b3d1-c872cf2b7c55 None admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:04.571361 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4f1670-2676-45a9-b3d1-c872cf2b7c55 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:04.571627 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4f1670-2676-45a9-b3d1-c872cf2b7c55 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:04.572395 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f4f1670-2676-45a9-b3d1-c872cf2b7c55 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:04.573797 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f4f1670-2676-45a9-b3d1-c872cf2b7c55 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:04.606539 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 359/706] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 08:49:04 2022] POST /identity/v3/endpoints => generated 347 bytes in 193 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 20 08:49:04.632676 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:04.650568 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12c2841a-68d8-4084-afeb-678bd0ea13b8 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:04.696964 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f5e42a-427c-4041-8b2f-c242c4d20ade None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:04.697576 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f5e42a-427c-4041-8b2f-c242c4d20ade None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:04.697850 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f5e42a-427c-4041-8b2f-c242c4d20ade None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:04.698116 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f5e42a-427c-4041-8b2f-c242c4d20ade None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:04.698873 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77f5e42a-427c-4041-8b2f-c242c4d20ade None admin] RBAC: Adding query filter params (role.id=37a51151d3c042ffbaca4dc2bf317baf, scope.project.id=d9857b610fb542969cfb5efbbdb8ce61, user.id=f4b0e9187ce74f7888e9e7094be5c824) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:49:04.699176 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77f5e42a-427c-4041-8b2f-c242c4d20ade None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:04.700703 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77f5e42a-427c-4041-8b2f-c242c4d20ade None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:04.707977 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 348/707] 10.222.0.36 () {56 vars in 1374 bytes} [Thu Oct 20 08:49:04 2022] GET /identity/v3/role_assignments?user.id=f4b0e9187ce74f7888e9e7094be5c824&scope.project.id=d9857b610fb542969cfb5efbbdb8ce61&role.id=37a51151d3c042ffbaca4dc2bf317baf => generated 608 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 08:49:04.935490 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67eb3acf-f1cd-46d2-b1e1-9fe483902e43 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:04.935882 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67eb3acf-f1cd-46d2-b1e1-9fe483902e43 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:04.936081 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67eb3acf-f1cd-46d2-b1e1-9fe483902e43 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:04.937665 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 360/708] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:49:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:49:04.943701 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b32f6d56-26e8-4c29-a92f-45f511956dcf None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:04.943986 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b32f6d56-26e8-4c29-a92f-45f511956dcf None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:04.944263 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b32f6d56-26e8-4c29-a92f-45f511956dcf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:04.963014 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-b32f6d56-26e8-4c29-a92f-45f511956dcf None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:49:04.964205 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-b32f6d56-26e8-4c29-a92f-45f511956dcf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:49:05.039790 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 349/709] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:49:04 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 20 08:49:05.306220 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67eb3acf-f1cd-46d2-b1e1-9fe483902e43 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:05.412962 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67eb3acf-f1cd-46d2-b1e1-9fe483902e43 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:05.446699 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b43827c6-e753-416b-a336-1003195d034f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:05.447326 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43827c6-e753-416b-a336-1003195d034f None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:05.447600 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43827c6-e753-416b-a336-1003195d034f None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:05.447866 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43827c6-e753-416b-a336-1003195d034f None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:05.448445 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b43827c6-e753-416b-a336-1003195d034f None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:05.449803 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b43827c6-e753-416b-a336-1003195d034f None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:05.459598 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-b43827c6-e753-416b-a336-1003195d034f None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 20 08:49:05.459598 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:49:05.459598 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 20 08:49:05.459598 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 20 08:49:05.459598 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 20 08:49:05.459598 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 20 08:49:05.459598 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 20 08:49:05.459598 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 20 08:49:05.459598 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 20 08:49:05.459598 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:49:05.459598 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 20 08:49:05.459598 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:49:05.459598 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:49:05.459598 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:49:05.459598 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:49:05.459598 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:49:05.459598 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:49:05.460593 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:49:05.460593 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:49:05.460593 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:49:05.460593 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:49:05.460593 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:49:05.460593 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:49:05.460593 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 20 08:49:05.460593 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 20 08:49:05.460593 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 20 08:49:05.460593 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 20 08:49:05.460593 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:49:05.460593 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:49:05.460593 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:49:05.460593 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:49:05.460593 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:49:05.460593 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:49:05.462062 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:49:05.462062 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:49:05.462062 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:49:05.462062 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:49:05.462062 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:49:05.462062 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:49:05.462062 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:49:05.462062 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:49:05.462062 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:49:05.462062 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:49:05.462062 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 20 08:49:05.462062 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 20 08:49:05.462062 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 20 08:49:05.462062 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:49:05.462062 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 361/710] 10.222.0.36 () {56 vars in 1126 bytes} [Thu Oct 20 08:49:05 2022] GET /identity/v3/services/object-store => generated 93 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:49:05.466718 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b32f6d56-26e8-4c29-a92f-45f511956dcf None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:05.548673 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:05.573743 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b32f6d56-26e8-4c29-a92f-45f511956dcf None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:05.615204 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff1df2ce-da3e-4e5c-a60d-d4499f372eec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:05.616161 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff1df2ce-da3e-4e5c-a60d-d4499f372eec None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:05.616652 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff1df2ce-da3e-4e5c-a60d-d4499f372eec None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:05.617133 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff1df2ce-da3e-4e5c-a60d-d4499f372eec None admin] PATH_INFO: `/v3/services` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:05.617944 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff1df2ce-da3e-4e5c-a60d-d4499f372eec None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:49:05.618503 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff1df2ce-da3e-4e5c-a60d-d4499f372eec None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:05.620067 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff1df2ce-da3e-4e5c-a60d-d4499f372eec None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:05.625358 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 350/711] 10.222.0.36 () {56 vars in 1122 bytes} [Thu Oct 20 08:49:05 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:49:05.630808 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b43827c6-e753-416b-a336-1003195d034f None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:05.650726 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:05.742684 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b43827c6-e753-416b-a336-1003195d034f None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:05.777660 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-626fd3ab-fe32-4a62-b0c2-18759f2494a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:05.778259 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626fd3ab-fe32-4a62-b0c2-18759f2494a4 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:05.778562 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626fd3ab-fe32-4a62-b0c2-18759f2494a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:05.778834 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626fd3ab-fe32-4a62-b0c2-18759f2494a4 None admin] PATH_INFO: `/v3/services` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:05.779456 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-626fd3ab-fe32-4a62-b0c2-18759f2494a4 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:49:05.779754 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-626fd3ab-fe32-4a62-b0c2-18759f2494a4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:05.781118 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-626fd3ab-fe32-4a62-b0c2-18759f2494a4 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:05.786397 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 362/712] 10.222.0.36 () {56 vars in 1122 bytes} [Thu Oct 20 08:49:05 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:49:05.792058 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff1df2ce-da3e-4e5c-a60d-d4499f372eec None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:05.893656 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff1df2ce-da3e-4e5c-a60d-d4499f372eec None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:05.930083 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63e7025b-7e90-4112-a9a5-096095b6a046 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:05.930712 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e7025b-7e90-4112-a9a5-096095b6a046 None admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:05.930992 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e7025b-7e90-4112-a9a5-096095b6a046 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:05.931260 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e7025b-7e90-4112-a9a5-096095b6a046 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:05.931931 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63e7025b-7e90-4112-a9a5-096095b6a046 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:05.933305 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63e7025b-7e90-4112-a9a5-096095b6a046 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:05.943445 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:05.957555 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 351/713] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 08:49:05 2022] POST /identity/v3/endpoints => generated 349 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 08:49:06.478710 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda62d93-f39f-4f2b-babb-18b5e8fc6654 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:06.479355 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda62d93-f39f-4f2b-babb-18b5e8fc6654 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:06.479838 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda62d93-f39f-4f2b-babb-18b5e8fc6654 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:06.481767 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 363/714] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:49:06 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:49:06.486892 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a5da5c-18c3-4a6a-82c8-7d1ad2421346 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:06.487179 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a5da5c-18c3-4a6a-82c8-7d1ad2421346 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:06.487458 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a5da5c-18c3-4a6a-82c8-7d1ad2421346 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:06.506883 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-62a5da5c-18c3-4a6a-82c8-7d1ad2421346 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:49:06.508078 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-62a5da5c-18c3-4a6a-82c8-7d1ad2421346 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:49:06.585471 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 352/715] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:49:06 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 20 08:49:06.857613 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dda62d93-f39f-4f2b-babb-18b5e8fc6654 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:06.893045 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad350dfb-d2c5-46ac-bb2f-17ac29fe14b8 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:06.893439 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad350dfb-d2c5-46ac-bb2f-17ac29fe14b8 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:06.893633 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad350dfb-d2c5-46ac-bb2f-17ac29fe14b8 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:06.895193 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 353/716] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:49:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:49:06.900205 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d76d07-3e06-42a1-aff7-fb05a8110b8a None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:06.900485 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d76d07-3e06-42a1-aff7-fb05a8110b8a None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:06.900756 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d76d07-3e06-42a1-aff7-fb05a8110b8a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:06.918829 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-17d76d07-3e06-42a1-aff7-fb05a8110b8a None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:49:06.920375 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-17d76d07-3e06-42a1-aff7-fb05a8110b8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:49:06.950355 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:06.972312 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dda62d93-f39f-4f2b-babb-18b5e8fc6654 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:07.016594 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70fd9dbb-5939-4213-8f4e-97ed1de9dab7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:07.017217 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70fd9dbb-5939-4213-8f4e-97ed1de9dab7 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:07.017488 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70fd9dbb-5939-4213-8f4e-97ed1de9dab7 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:07.017749 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70fd9dbb-5939-4213-8f4e-97ed1de9dab7 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:07.018317 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70fd9dbb-5939-4213-8f4e-97ed1de9dab7 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:07.022779 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 354/717] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:49:06 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 20 08:49:07.026512 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70fd9dbb-5939-4213-8f4e-97ed1de9dab7 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:07.036618 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-70fd9dbb-5939-4213-8f4e-97ed1de9dab7 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 20 08:49:07.036618 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:49:07.036618 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 20 08:49:07.036618 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 20 08:49:07.036618 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 20 08:49:07.036618 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 20 08:49:07.036618 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 20 08:49:07.036618 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 20 08:49:07.036618 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 20 08:49:07.036618 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:49:07.036618 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 20 08:49:07.036618 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:49:07.036618 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:49:07.036618 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:49:07.036618 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:49:07.036618 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:49:07.036618 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:49:07.037724 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:49:07.037724 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:49:07.037724 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:49:07.037724 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:49:07.037724 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:49:07.037724 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:49:07.037724 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 20 08:49:07.037724 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 20 08:49:07.037724 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 20 08:49:07.037724 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 20 08:49:07.037724 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:49:07.037724 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:49:07.037724 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:49:07.037724 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:49:07.037724 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:49:07.037724 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:49:07.038627 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:49:07.038627 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:49:07.038627 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:49:07.038627 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:49:07.038627 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:49:07.038627 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:49:07.038627 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:49:07.038627 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:49:07.038627 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:49:07.038627 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:49:07.038627 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 20 08:49:07.038627 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 20 08:49:07.038627 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 20 08:49:07.038627 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:49:07.042203 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 364/718] 10.222.0.36 () {56 vars in 1114 bytes} [Thu Oct 20 08:49:06 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:49:07.047642 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17d76d07-3e06-42a1-aff7-fb05a8110b8a None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:07.052261 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4dbc180-783d-44d5-ae79-51f119563495 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:07.052543 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4dbc180-783d-44d5-ae79-51f119563495 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:07.052809 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4dbc180-783d-44d5-ae79-51f119563495 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:07.054121 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 365/719] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:49:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 20 08:49:07.059077 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b85dce0-0002-40b8-ac76-96bd7c383781 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:07.059350 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b85dce0-0002-40b8-ac76-96bd7c383781 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:07.059617 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b85dce0-0002-40b8-ac76-96bd7c383781 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:07.079820 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-0b85dce0-0002-40b8-ac76-96bd7c383781 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:49:07.081356 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-0b85dce0-0002-40b8-ac76-96bd7c383781 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:49:07.111262 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:07.161626 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 366/720] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:49:07 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 20 08:49:07.173339 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17d76d07-3e06-42a1-aff7-fb05a8110b8a None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:07.211201 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01040f5a-0257-4dea-a8ca-4e6d3c5a8593 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:07.211805 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01040f5a-0257-4dea-a8ca-4e6d3c5a8593 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:07.212080 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01040f5a-0257-4dea-a8ca-4e6d3c5a8593 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:07.212343 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01040f5a-0257-4dea-a8ca-4e6d3c5a8593 None admin] PATH_INFO: `/v3/services` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:07.212964 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01040f5a-0257-4dea-a8ca-4e6d3c5a8593 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:49:07.213263 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01040f5a-0257-4dea-a8ca-4e6d3c5a8593 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:07.214691 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01040f5a-0257-4dea-a8ca-4e6d3c5a8593 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:07.219915 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 355/721] 10.222.0.36 () {56 vars in 1114 bytes} [Thu Oct 20 08:49:07 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:49:07.227564 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b85dce0-0002-40b8-ac76-96bd7c383781 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:07.305540 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:07.316488 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01040f5a-0257-4dea-a8ca-4e6d3c5a8593 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:07.331673 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b85dce0-0002-40b8-ac76-96bd7c383781 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:07.370167 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd0b4de6-17c4-4758-8ee7-292a57b89f5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:07.370678 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0b4de6-17c4-4758-8ee7-292a57b89f5c None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:07.370956 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0b4de6-17c4-4758-8ee7-292a57b89f5c None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:07.371227 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0b4de6-17c4-4758-8ee7-292a57b89f5c None admin] PATH_INFO: `/v3/services` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:07.371851 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd0b4de6-17c4-4758-8ee7-292a57b89f5c None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:49:07.372145 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd0b4de6-17c4-4758-8ee7-292a57b89f5c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:07.373627 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd0b4de6-17c4-4758-8ee7-292a57b89f5c None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:07.378974 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 367/722] 10.222.0.36 () {56 vars in 1114 bytes} [Thu Oct 20 08:49:07 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:49:07.386552 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd0b4de6-17c4-4758-8ee7-292a57b89f5c None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:07.442052 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01040f5a-0257-4dea-a8ca-4e6d3c5a8593 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:07.461015 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:07.482948 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1579d7f4-1091-49e2-b7fe-b8fa04869df0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:07.483580 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1579d7f4-1091-49e2-b7fe-b8fa04869df0 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:07.483853 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1579d7f4-1091-49e2-b7fe-b8fa04869df0 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:07.484122 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1579d7f4-1091-49e2-b7fe-b8fa04869df0 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:07.484700 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1579d7f4-1091-49e2-b7fe-b8fa04869df0 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:07.486069 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1579d7f4-1091-49e2-b7fe-b8fa04869df0 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:07.494754 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-1579d7f4-1091-49e2-b7fe-b8fa04869df0 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Oct 20 08:49:07.494754 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:49:07.494754 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 20 08:49:07.494754 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 20 08:49:07.494754 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 20 08:49:07.494754 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 20 08:49:07.494754 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 20 08:49:07.494754 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 20 08:49:07.494754 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 20 08:49:07.494754 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:49:07.494754 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 20 08:49:07.494754 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:49:07.494754 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:49:07.494754 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:49:07.494754 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:49:07.494754 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:49:07.494754 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:49:07.495726 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:49:07.495726 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:49:07.495726 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:49:07.495726 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:49:07.495726 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:49:07.495726 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:49:07.495726 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 20 08:49:07.495726 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 20 08:49:07.495726 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 20 08:49:07.495726 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 20 08:49:07.495726 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:49:07.495726 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:49:07.495726 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:49:07.495726 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:49:07.495726 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:49:07.495726 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:49:07.496572 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:49:07.496572 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:49:07.496572 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:49:07.496572 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:49:07.496572 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:49:07.496572 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:49:07.496572 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:49:07.496572 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:49:07.496572 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:49:07.496572 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:49:07.496572 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 20 08:49:07.496572 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 20 08:49:07.496572 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 20 08:49:07.496572 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:49:07.496572 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 356/723] 10.222.0.36 () {56 vars in 1111 bytes} [Thu Oct 20 08:49:07 2022] GET /identity/v3/services/compute => generated 88 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 20 08:49:07.497384 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1579d7f4-1091-49e2-b7fe-b8fa04869df0 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:07.516970 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd0b4de6-17c4-4758-8ee7-292a57b89f5c None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:07.559726 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e5333c8-dd8c-4739-b16c-27cd2c278845 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:07.560317 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5333c8-dd8c-4739-b16c-27cd2c278845 None admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:07.560595 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5333c8-dd8c-4739-b16c-27cd2c278845 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:07.561053 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5333c8-dd8c-4739-b16c-27cd2c278845 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:07.561555 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e5333c8-dd8c-4739-b16c-27cd2c278845 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:07.562992 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e5333c8-dd8c-4739-b16c-27cd2c278845 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:07.572116 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:07.594505 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 368/724] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 08:49:07 2022] POST /identity/v3/endpoints => generated 346 bytes in 198 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 08:49:07.594505 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e5333c8-dd8c-4739-b16c-27cd2c278845 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:07.615048 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1579d7f4-1091-49e2-b7fe-b8fa04869df0 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:07.668934 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34fca0eb-550b-4337-ab66-17c3c27c7c60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:07.669403 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34fca0eb-550b-4337-ab66-17c3c27c7c60 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:07.669679 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34fca0eb-550b-4337-ab66-17c3c27c7c60 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:07.669997 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34fca0eb-550b-4337-ab66-17c3c27c7c60 None admin] PATH_INFO: `/v3/services/image` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:07.670626 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34fca0eb-550b-4337-ab66-17c3c27c7c60 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:07.672102 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34fca0eb-550b-4337-ab66-17c3c27c7c60 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:07.682487 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-34fca0eb-550b-4337-ab66-17c3c27c7c60 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Oct 20 08:49:07.682487 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:49:07.682487 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 20 08:49:07.682487 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 20 08:49:07.682487 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 20 08:49:07.682487 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 20 08:49:07.682487 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 20 08:49:07.682487 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 20 08:49:07.682487 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 20 08:49:07.682487 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:49:07.682487 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 20 08:49:07.682487 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:49:07.682487 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:49:07.682487 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:49:07.682487 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:49:07.682487 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:49:07.682487 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:49:07.683627 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:49:07.683627 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:49:07.683627 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:49:07.683627 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:49:07.683627 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:49:07.683627 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:49:07.683627 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 20 08:49:07.683627 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 20 08:49:07.683627 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 20 08:49:07.683627 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 20 08:49:07.683627 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:49:07.683627 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:49:07.683627 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:49:07.683627 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:49:07.683627 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:49:07.683627 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:49:07.684476 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:49:07.684476 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:49:07.684476 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:49:07.684476 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:49:07.684476 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:49:07.684476 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:49:07.684476 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:49:07.684476 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:49:07.684476 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:49:07.684476 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:49:07.684476 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 20 08:49:07.684476 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 20 08:49:07.684476 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 20 08:49:07.684476 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:49:07.689007 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 357/725] 10.222.0.36 () {56 vars in 1105 bytes} [Thu Oct 20 08:49:07 2022] GET /identity/v3/services/image => generated 86 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:49:07.690092 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34fca0eb-550b-4337-ab66-17c3c27c7c60 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:07.731104 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e5333c8-dd8c-4739-b16c-27cd2c278845 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:07.747091 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:07.772647 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8067f22e-fb3d-465a-8c13-becfb9cf5eb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:07.773819 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8067f22e-fb3d-465a-8c13-becfb9cf5eb5 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:07.774307 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8067f22e-fb3d-465a-8c13-becfb9cf5eb5 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:07.774826 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8067f22e-fb3d-465a-8c13-becfb9cf5eb5 None admin] PATH_INFO: `/v3/services` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:07.775668 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8067f22e-fb3d-465a-8c13-becfb9cf5eb5 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:49:07.776200 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8067f22e-fb3d-465a-8c13-becfb9cf5eb5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:07.777881 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8067f22e-fb3d-465a-8c13-becfb9cf5eb5 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:07.785823 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 369/726] 10.222.0.36 () {56 vars in 1112 bytes} [Thu Oct 20 08:49:07 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:49:07.793111 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8067f22e-fb3d-465a-8c13-becfb9cf5eb5 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:07.815377 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34fca0eb-550b-4337-ab66-17c3c27c7c60 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:07.854456 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e792015e-dae2-4768-94f3-9ead8620e92f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:07.855090 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e792015e-dae2-4768-94f3-9ead8620e92f None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:07.855991 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e792015e-dae2-4768-94f3-9ead8620e92f None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:07.855991 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e792015e-dae2-4768-94f3-9ead8620e92f None admin] PATH_INFO: `/v3/services` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:07.856249 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e792015e-dae2-4768-94f3-9ead8620e92f None admin] RBAC: Adding query filter params (name=image) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:49:07.856615 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e792015e-dae2-4768-94f3-9ead8620e92f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:07.857990 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e792015e-dae2-4768-94f3-9ead8620e92f None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:07.869252 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 358/727] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:49:07 2022] GET /identity/v3/services?name=image => generated 122 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:49:07.871952 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:07.875434 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e792015e-dae2-4768-94f3-9ead8620e92f None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:07.922171 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8067f22e-fb3d-465a-8c13-becfb9cf5eb5 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:07.962726 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfc0b125-a43d-41bf-a03c-c338b4895798 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:07.963332 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfc0b125-a43d-41bf-a03c-c338b4895798 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:07.963607 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfc0b125-a43d-41bf-a03c-c338b4895798 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:07.963875 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfc0b125-a43d-41bf-a03c-c338b4895798 None admin] PATH_INFO: `/v3/services` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:07.964538 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfc0b125-a43d-41bf-a03c-c338b4895798 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:49:07.965278 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfc0b125-a43d-41bf-a03c-c338b4895798 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:07.967076 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfc0b125-a43d-41bf-a03c-c338b4895798 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:07.972452 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 370/728] 10.222.0.36 () {56 vars in 1112 bytes} [Thu Oct 20 08:49:07 2022] GET /identity/v3/services?type=compute => generated 124 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 08:49:07.995708 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e792015e-dae2-4768-94f3-9ead8620e92f None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:07.999418 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:08.035556 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fc37d2e-7e59-49fc-b299-0b664242fe7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:08.036992 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc37d2e-7e59-49fc-b299-0b664242fe7a None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:08.037443 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc37d2e-7e59-49fc-b299-0b664242fe7a None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:08.037865 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc37d2e-7e59-49fc-b299-0b664242fe7a None admin] PATH_INFO: `/v3/services` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:08.038725 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fc37d2e-7e59-49fc-b299-0b664242fe7a None admin] RBAC: Adding query filter params (type=image) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:49:08.039182 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fc37d2e-7e59-49fc-b299-0b664242fe7a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:08.040773 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fc37d2e-7e59-49fc-b299-0b664242fe7a None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:08.047031 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 359/729] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:49:07 2022] GET /identity/v3/services?type=image => generated 122 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:49:08.173921 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04290a87-792d-4cb7-ad2f-938f887759f5 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:08.174133 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04290a87-792d-4cb7-ad2f-938f887759f5 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:08.174411 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04290a87-792d-4cb7-ad2f-938f887759f5 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:08.176117 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 371/730] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:49:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:49:08.182610 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59604fe2-7518-44a3-8eb3-6c8edeb35ff2 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:08.182934 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59604fe2-7518-44a3-8eb3-6c8edeb35ff2 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:08.183177 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59604fe2-7518-44a3-8eb3-6c8edeb35ff2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:08.207186 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-59604fe2-7518-44a3-8eb3-6c8edeb35ff2 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:49:08.208472 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-59604fe2-7518-44a3-8eb3-6c8edeb35ff2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:49:08.286729 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 360/731] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:49:08 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 20 08:49:08.556531 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04290a87-792d-4cb7-ad2f-938f887759f5 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:08.670503 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04290a87-792d-4cb7-ad2f-938f887759f5 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:08.694806 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:08.705978 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0846125-f9fb-4d74-8814-f3a8f0266538 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:08.707062 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0846125-f9fb-4d74-8814-f3a8f0266538 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:08.707770 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0846125-f9fb-4d74-8814-f3a8f0266538 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:08.708610 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0846125-f9fb-4d74-8814-f3a8f0266538 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:08.709451 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0846125-f9fb-4d74-8814-f3a8f0266538 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:08.710912 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0846125-f9fb-4d74-8814-f3a8f0266538 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:08.720112 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-c0846125-f9fb-4d74-8814-f3a8f0266538 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 20 08:49:08.720112 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:49:08.720112 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 20 08:49:08.720112 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 20 08:49:08.720112 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 20 08:49:08.720112 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 20 08:49:08.720112 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 20 08:49:08.720112 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 20 08:49:08.720112 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 20 08:49:08.720112 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:49:08.720112 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 20 08:49:08.720112 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:49:08.720112 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:49:08.720112 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:49:08.720112 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:49:08.720112 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:49:08.720112 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:49:08.721100 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:49:08.721100 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:49:08.721100 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:49:08.721100 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:49:08.721100 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:49:08.721100 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:49:08.721100 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 20 08:49:08.721100 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 20 08:49:08.721100 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 20 08:49:08.721100 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 20 08:49:08.721100 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:49:08.721100 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:49:08.721100 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:49:08.721100 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:49:08.721100 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:49:08.721100 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:49:08.721948 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:49:08.721948 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:49:08.721948 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:49:08.721948 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:49:08.721948 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:49:08.721948 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:49:08.721948 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:49:08.721948 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:49:08.721948 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:49:08.721948 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:49:08.721948 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 20 08:49:08.721948 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 20 08:49:08.721948 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 20 08:49:08.721948 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:49:08.721948 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 372/732] 10.222.0.36 () {56 vars in 1126 bytes} [Thu Oct 20 08:49:08 2022] GET /identity/v3/services/object-store => generated 93 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:49:08.729401 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59604fe2-7518-44a3-8eb3-6c8edeb35ff2 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:08.827033 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:08.831006 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59604fe2-7518-44a3-8eb3-6c8edeb35ff2 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:08.866563 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae76f6e5-c28e-4dbd-a56b-4ec3a14d1cd1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:08.867163 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae76f6e5-c28e-4dbd-a56b-4ec3a14d1cd1 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:08.867439 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae76f6e5-c28e-4dbd-a56b-4ec3a14d1cd1 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:08.867702 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae76f6e5-c28e-4dbd-a56b-4ec3a14d1cd1 None admin] PATH_INFO: `/v3/services` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:08.868314 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae76f6e5-c28e-4dbd-a56b-4ec3a14d1cd1 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:49:08.868609 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae76f6e5-c28e-4dbd-a56b-4ec3a14d1cd1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:08.869968 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae76f6e5-c28e-4dbd-a56b-4ec3a14d1cd1 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:08.875117 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 361/733] 10.222.0.36 () {56 vars in 1122 bytes} [Thu Oct 20 08:49:08 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:49:08.881586 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0846125-f9fb-4d74-8814-f3a8f0266538 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:08.994700 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0846125-f9fb-4d74-8814-f3a8f0266538 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:09.033188 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ecadde0-c7d3-441a-b1dd-c46622a34fd0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:09.033796 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ecadde0-c7d3-441a-b1dd-c46622a34fd0 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:09.034159 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ecadde0-c7d3-441a-b1dd-c46622a34fd0 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:09.034435 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ecadde0-c7d3-441a-b1dd-c46622a34fd0 None admin] PATH_INFO: `/v3/services` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:09.035088 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ecadde0-c7d3-441a-b1dd-c46622a34fd0 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:49:09.035391 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ecadde0-c7d3-441a-b1dd-c46622a34fd0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:09.036786 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ecadde0-c7d3-441a-b1dd-c46622a34fd0 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:09.042249 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 373/734] 10.222.0.36 () {56 vars in 1122 bytes} [Thu Oct 20 08:49:08 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:49:09.048031 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae76f6e5-c28e-4dbd-a56b-4ec3a14d1cd1 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:09.155118 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae76f6e5-c28e-4dbd-a56b-4ec3a14d1cd1 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:09.189021 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41e2c865-be30-4bf7-90c9-4533c8ad7c4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:09.189624 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e2c865-be30-4bf7-90c9-4533c8ad7c4b None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:09.189898 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e2c865-be30-4bf7-90c9-4533c8ad7c4b None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:09.190167 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e2c865-be30-4bf7-90c9-4533c8ad7c4b None admin] PATH_INFO: `/v3/endpoints` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:09.190888 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41e2c865-be30-4bf7-90c9-4533c8ad7c4b None admin] RBAC: Adding query filter params (interface=admin, service_id=23da7943ed384de8bd44e33228d6ccc2, region_id=RegionOne) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:49:09.191188 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41e2c865-be30-4bf7-90c9-4533c8ad7c4b None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:09.193150 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41e2c865-be30-4bf7-90c9-4533c8ad7c4b None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:09.198566 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 362/735] 10.222.0.36 () {56 vars in 1249 bytes} [Thu Oct 20 08:49:09 2022] GET /identity/v3/endpoints?service_id=23da7943ed384de8bd44e33228d6ccc2&interface=admin®ion_id=RegionOne => generated 193 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:49:09.204216 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ecadde0-c7d3-441a-b1dd-c46622a34fd0 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:09.262811 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:09.306770 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ecadde0-c7d3-441a-b1dd-c46622a34fd0 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:09.342626 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ea39776-1e65-4069-9aa9-98451605822c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:09.343236 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea39776-1e65-4069-9aa9-98451605822c None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:09.343514 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea39776-1e65-4069-9aa9-98451605822c None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:09.343786 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea39776-1e65-4069-9aa9-98451605822c None admin] PATH_INFO: `/v3/services` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:09.344392 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ea39776-1e65-4069-9aa9-98451605822c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:09.345750 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ea39776-1e65-4069-9aa9-98451605822c None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:09.352120 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 374/736] 10.222.0.36 () {56 vars in 1087 bytes} [Thu Oct 20 08:49:09 2022] GET /identity/v3/services => generated 1497 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 08:49:09.910284 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16754a4b-f793-4a58-80a7-b6d04aae4f77 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:09.910829 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16754a4b-f793-4a58-80a7-b6d04aae4f77 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:09.910829 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16754a4b-f793-4a58-80a7-b6d04aae4f77 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:09.912519 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 363/737] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:49:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:49:09.917813 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8916d54-9517-411c-834d-c92f30610f59 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:09.918239 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8916d54-9517-411c-834d-c92f30610f59 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:09.918427 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8916d54-9517-411c-834d-c92f30610f59 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:09.937141 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-c8916d54-9517-411c-834d-c92f30610f59 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:49:09.938309 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-c8916d54-9517-411c-834d-c92f30610f59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:49:10.010608 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 375/738] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:49:09 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 20 08:49:10.269721 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a35d30-20fe-4476-931a-fa3d372ef541 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:10.270510 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a35d30-20fe-4476-931a-fa3d372ef541 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:10.271031 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a35d30-20fe-4476-931a-fa3d372ef541 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:10.272842 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 364/739] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:49:10 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:49:10.275152 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8916d54-9517-411c-834d-c92f30610f59 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:10.287604 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac87397a-e646-40c3-8f5e-90b624b25a1b None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:10.288157 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac87397a-e646-40c3-8f5e-90b624b25a1b None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:10.288641 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac87397a-e646-40c3-8f5e-90b624b25a1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:10.308244 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-ac87397a-e646-40c3-8f5e-90b624b25a1b None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:49:10.309690 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-ac87397a-e646-40c3-8f5e-90b624b25a1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:49:10.323623 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:10.334361 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:10.385746 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8916d54-9517-411c-834d-c92f30610f59 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:10.399815 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 365/740] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:49:10 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 20 08:49:10.402102 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef2df305-4c46-41f2-b74a-f17f3c9ef0d7 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:10.402888 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef2df305-4c46-41f2-b74a-f17f3c9ef0d7 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:10.403405 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef2df305-4c46-41f2-b74a-f17f3c9ef0d7 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:10.409517 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 366/741] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:49:10 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:49:10.412152 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad3bf5d-ca14-41ba-8328-20e32de42bc8 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:10.412650 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad3bf5d-ca14-41ba-8328-20e32de42bc8 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:10.413381 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad3bf5d-ca14-41ba-8328-20e32de42bc8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:10.424852 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fa48de8-9679-4e78-b5bb-920459e95721 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:10.425692 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa48de8-9679-4e78-b5bb-920459e95721 None admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:10.426209 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa48de8-9679-4e78-b5bb-920459e95721 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:10.426712 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa48de8-9679-4e78-b5bb-920459e95721 None admin] PATH_INFO: `/v3/projects` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:10.428001 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fa48de8-9679-4e78-b5bb-920459e95721 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:10.429567 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fa48de8-9679-4e78-b5bb-920459e95721 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:10.432328 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: WARNING keystone.server.flask.common [None req-4fa48de8-9679-4e78-b5bb-920459e95721 None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Oct 20 08:49:10.436164 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-4ad3bf5d-ca14-41ba-8328-20e32de42bc8 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:49:10.437901 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-4ad3bf5d-ca14-41ba-8328-20e32de42bc8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:49:10.452549 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.sql.core [None req-4fa48de8-9679-4e78-b5bb-920459e95721 None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Oct 20 08:49:10.452549 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 20 08:49:10.452549 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [parameters: {'id': '0015515673e7453e996af10269070007', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Oct 20 08:49:10.452549 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81562) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 20 08:49:10.461628 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-4fa48de8-9679-4e78-b5bb-920459e95721 None admin] Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 0015515673e7453e996af10269070007.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 0015515673e7453e996af10269070007. Oct 20 08:49:10.461628 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:49:10.461628 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 20 08:49:10.461628 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 20 08:49:10.461628 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 20 08:49:10.461628 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 20 08:49:10.461628 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 20 08:49:10.461628 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application result = self._query(query) Oct 20 08:49:10.461628 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 20 08:49:10.461628 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application conn.query(q) Oct 20 08:49:10.461628 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 20 08:49:10.461628 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 20 08:49:10.461628 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 20 08:49:10.462606 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application result.read() Oct 20 08:49:10.462606 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 20 08:49:10.462606 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 20 08:49:10.462606 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 20 08:49:10.462606 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 20 08:49:10.462606 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 20 08:49:10.462606 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 20 08:49:10.462606 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 20 08:49:10.462606 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 20 08:49:10.462606 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Oct 20 08:49:10.462606 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:49:10.462606 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 20 08:49:10.462606 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:49:10.462606 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:49:10.462606 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 20 08:49:10.462606 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 20 08:49:10.462606 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Oct 20 08:49:10.462606 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return project_ref.to_dict() Oct 20 08:49:10.463838 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 20 08:49:10.463838 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application next(self.gen) Oct 20 08:49:10.463838 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Oct 20 08:49:10.463838 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application yield resource Oct 20 08:49:10.463838 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 20 08:49:10.463838 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application next(self.gen) Oct 20 08:49:10.463838 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Oct 20 08:49:10.463838 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application self.session.rollback() Oct 20 08:49:10.463838 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 20 08:49:10.463838 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application self.force_reraise() Oct 20 08:49:10.463838 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 20 08:49:10.463838 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise self.value Oct 20 08:49:10.463838 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Oct 20 08:49:10.463838 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 20 08:49:10.463838 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Oct 20 08:49:10.463838 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application session.commit() Oct 20 08:49:10.463838 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 20 08:49:10.465019 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 20 08:49:10.465019 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 20 08:49:10.465019 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application self._prepare_impl() Oct 20 08:49:10.465019 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 20 08:49:10.465019 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application self.session.flush() Oct 20 08:49:10.465019 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 20 08:49:10.465019 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application self._flush(objects) Oct 20 08:49:10.465019 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 20 08:49:10.465019 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 20 08:49:10.465019 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 20 08:49:10.465019 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application compat.raise_( Oct 20 08:49:10.465019 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 20 08:49:10.465019 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception Oct 20 08:49:10.465019 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 20 08:49:10.465019 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application flush_context.execute() Oct 20 08:49:10.465019 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 20 08:49:10.465019 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rec.execute(self) Oct 20 08:49:10.466153 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 20 08:49:10.466153 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 20 08:49:10.466153 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 20 08:49:10.466153 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 20 08:49:10.466153 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Oct 20 08:49:10.466153 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application c = connection._execute_20( Oct 20 08:49:10.466153 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 20 08:49:10.466153 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 20 08:49:10.466153 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 20 08:49:10.466153 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 20 08:49:10.466153 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 20 08:49:10.466153 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 20 08:49:10.466153 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 20 08:49:10.466153 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 20 08:49:10.466153 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 20 08:49:10.467165 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 20 08:49:10.467165 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 20 08:49:10.467165 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception Oct 20 08:49:10.467165 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 20 08:49:10.467165 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 20 08:49:10.467165 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 20 08:49:10.467165 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 20 08:49:10.467165 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 20 08:49:10.467165 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application result = self._query(query) Oct 20 08:49:10.467165 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 20 08:49:10.467165 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application conn.query(q) Oct 20 08:49:10.467165 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 20 08:49:10.467165 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 20 08:49:10.467165 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 20 08:49:10.467165 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application result.read() Oct 20 08:49:10.467165 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 20 08:49:10.467165 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 20 08:49:10.468644 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 20 08:49:10.468644 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 20 08:49:10.468644 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 20 08:49:10.468644 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 20 08:49:10.468644 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 20 08:49:10.468644 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 20 08:49:10.468644 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Oct 20 08:49:10.468644 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 20 08:49:10.468644 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [parameters: {'id': '0015515673e7453e996af10269070007', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Oct 20 08:49:10.468644 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 20 08:49:10.468644 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:49:10.468644 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 20 08:49:10.468644 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:49:10.468644 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:49:10.468644 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Oct 20 08:49:10.470276 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Oct 20 08:49:10.470276 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 20 08:49:10.470276 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 20 08:49:10.470276 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name service at domain ID default. Oct 20 08:49:10.470276 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:49:10.470276 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 20 08:49:10.470276 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:49:10.470276 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:49:10.470276 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:49:10.470276 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:49:10.470276 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:49:10.470276 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:49:10.470276 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:49:10.470276 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:49:10.470276 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:49:10.470276 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:49:10.470276 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:49:10.472284 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:49:10.472284 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 176, in post Oct 20 08:49:10.472284 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_project( Oct 20 08:49:10.472284 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:49:10.472284 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:49:10.472284 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Oct 20 08:49:10.472284 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.Conflict( Oct 20 08:49:10.472284 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 0015515673e7453e996af10269070007. Oct 20 08:49:10.472284 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:49:10.475153 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 376/742] 10.222.0.36 () {60 vars in 1140 bytes} [Thu Oct 20 08:49:10 2022] POST /identity/v3/projects => generated 262 bytes in 199 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:49:10.487813 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fa48de8-9679-4e78-b5bb-920459e95721 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:10.519521 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 367/743] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:49:10 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 20 08:49:10.599755 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:10.602880 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fa48de8-9679-4e78-b5bb-920459e95721 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:10.638424 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1c2b12-53e7-4315-a9f2-66983fd713a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:10.639070 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1c2b12-53e7-4315-a9f2-66983fd713a9 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:10.639343 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1c2b12-53e7-4315-a9f2-66983fd713a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:10.639662 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1c2b12-53e7-4315-a9f2-66983fd713a9 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:10.647011 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca1c2b12-53e7-4315-a9f2-66983fd713a9 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:10.648638 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca1c2b12-53e7-4315-a9f2-66983fd713a9 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:10.659795 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-ca1c2b12-53e7-4315-a9f2-66983fd713a9 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 20 08:49:10.659795 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:49:10.659795 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:49:10.659795 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:49:10.659795 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:49:10.659795 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:49:10.659795 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:49:10.659795 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:49:10.659795 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:49:10.659795 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:49:10.659795 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:49:10.659795 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:49:10.659795 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 20 08:49:10.659795 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 20 08:49:10.659795 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 20 08:49:10.662148 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 20 08:49:10.662148 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:49:10.662148 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:49:10.662148 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:49:10.662148 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:49:10.662148 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:49:10.662148 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:49:10.662148 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:49:10.662148 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:49:10.662148 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:49:10.662148 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:49:10.662148 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:49:10.662148 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:49:10.662148 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:49:10.662148 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:49:10.662148 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:49:10.662148 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:49:10.663362 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 20 08:49:10.663362 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 20 08:49:10.663362 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 20 08:49:10.663362 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 20 08:49:10.663362 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 20 08:49:10.663362 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 20 08:49:10.663362 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 20 08:49:10.663362 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:49:10.663362 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 377/744] 10.222.0.36 () {56 vars in 1111 bytes} [Thu Oct 20 08:49:10 2022] GET /identity/v3/projects/service => generated 88 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:49:10.667299 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ad3bf5d-ca14-41ba-8328-20e32de42bc8 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:10.685427 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1c2b12-53e7-4315-a9f2-66983fd713a9 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:10.775991 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:10.792645 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1c2b12-53e7-4315-a9f2-66983fd713a9 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:10.796072 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ad3bf5d-ca14-41ba-8328-20e32de42bc8 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:10.829237 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb3e82b-91db-45bc-a0b5-02c6e797f838 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:10.829840 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb3e82b-91db-45bc-a0b5-02c6e797f838 None admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:10.830140 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb3e82b-91db-45bc-a0b5-02c6e797f838 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:10.830412 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb3e82b-91db-45bc-a0b5-02c6e797f838 None admin] PATH_INFO: `/v3/services` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:10.831135 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffb3e82b-91db-45bc-a0b5-02c6e797f838 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:10.832524 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffb3e82b-91db-45bc-a0b5-02c6e797f838 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:10.838053 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b76332f-05bf-4738-954d-bde1259dafe8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:10.838720 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b76332f-05bf-4738-954d-bde1259dafe8 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:10.839091 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b76332f-05bf-4738-954d-bde1259dafe8 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:10.839349 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b76332f-05bf-4738-954d-bde1259dafe8 None admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:10.840057 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b76332f-05bf-4738-954d-bde1259dafe8 None admin] RBAC: Adding query filter params (name=service) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:49:10.840355 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b76332f-05bf-4738-954d-bde1259dafe8 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:10.842063 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b76332f-05bf-4738-954d-bde1259dafe8 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:10.852762 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 378/745] 10.222.0.36 () {60 vars in 1141 bytes} [Thu Oct 20 08:49:10 2022] POST /identity/v3/services => generated 242 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 20 08:49:10.853861 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb3e82b-91db-45bc-a0b5-02c6e797f838 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:10.865078 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 368/746] 10.222.0.36 () {56 vars in 1112 bytes} [Thu Oct 20 08:49:10 2022] GET /identity/v3/projects?name=service => generated 411 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:49:10.897795 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:10.989936 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb3e82b-91db-45bc-a0b5-02c6e797f838 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:11.030162 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-740942a7-115b-4781-ab1d-01b547cdcf94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:11.030877 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740942a7-115b-4781-ab1d-01b547cdcf94 None admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:11.031443 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740942a7-115b-4781-ab1d-01b547cdcf94 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:11.031710 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740942a7-115b-4781-ab1d-01b547cdcf94 None admin] PATH_INFO: `/v3/services` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:11.032397 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-740942a7-115b-4781-ab1d-01b547cdcf94 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:11.033946 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-740942a7-115b-4781-ab1d-01b547cdcf94 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:11.053455 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 379/747] 10.222.0.36 () {60 vars in 1141 bytes} [Thu Oct 20 08:49:10 2022] POST /identity/v3/services => generated 242 bytes in 200 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 08:49:11.555701 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd480b6b-e89d-491c-a54f-e10a7ac8d7b2 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:11.556216 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd480b6b-e89d-491c-a54f-e10a7ac8d7b2 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:11.556293 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd480b6b-e89d-491c-a54f-e10a7ac8d7b2 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:11.557817 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 369/748] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:49:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:49:11.563414 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37892352-33f8-4eb1-83cd-f0e89dadaedb None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:11.563702 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37892352-33f8-4eb1-83cd-f0e89dadaedb None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:11.563978 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37892352-33f8-4eb1-83cd-f0e89dadaedb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:11.582527 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-37892352-33f8-4eb1-83cd-f0e89dadaedb None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:49:11.584805 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-37892352-33f8-4eb1-83cd-f0e89dadaedb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:49:11.663747 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 380/749] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:49:11 2022] POST /identity/v3/auth/tokens => generated 1666 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 20 08:49:11.932688 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd480b6b-e89d-491c-a54f-e10a7ac8d7b2 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:12.040965 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd480b6b-e89d-491c-a54f-e10a7ac8d7b2 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:12.077414 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17d01690-8e16-4871-b6b3-d1c539c4b256 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:12.078350 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d01690-8e16-4871-b6b3-d1c539c4b256 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:12.078859 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d01690-8e16-4871-b6b3-d1c539c4b256 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:12.079435 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d01690-8e16-4871-b6b3-d1c539c4b256 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:12.080293 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17d01690-8e16-4871-b6b3-d1c539c4b256 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:12.081860 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17d01690-8e16-4871-b6b3-d1c539c4b256 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:12.091405 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-17d01690-8e16-4871-b6b3-d1c539c4b256 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 20 08:49:12.091405 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:49:12.091405 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 20 08:49:12.091405 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 20 08:49:12.091405 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 20 08:49:12.091405 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 20 08:49:12.091405 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 20 08:49:12.091405 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 20 08:49:12.091405 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 20 08:49:12.091405 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:49:12.091405 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 20 08:49:12.091405 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:49:12.091405 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:49:12.091405 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:49:12.091405 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:49:12.091405 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:49:12.091405 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:49:12.092412 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:49:12.092412 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:49:12.092412 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:49:12.092412 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:49:12.092412 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:49:12.092412 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:49:12.092412 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 20 08:49:12.092412 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 20 08:49:12.092412 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 20 08:49:12.092412 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 20 08:49:12.092412 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:49:12.092412 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:49:12.092412 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:49:12.092412 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:49:12.092412 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:49:12.092412 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:49:12.093264 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:49:12.093264 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:49:12.093264 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:49:12.093264 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:49:12.093264 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:49:12.093264 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:49:12.093264 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:49:12.093264 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:49:12.093264 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:49:12.093264 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:49:12.093264 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 20 08:49:12.093264 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 20 08:49:12.093264 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 20 08:49:12.093264 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:49:12.096004 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 370/750] 10.222.0.36 () {56 vars in 1126 bytes} [Thu Oct 20 08:49:11 2022] GET /identity/v3/services/object-store => generated 93 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:49:12.102498 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37892352-33f8-4eb1-83cd-f0e89dadaedb None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:12.103846 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:12.205906 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37892352-33f8-4eb1-83cd-f0e89dadaedb None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:12.245243 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db7c4b65-e0d3-49eb-8680-fb8e2ca0296c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:12.245854 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db7c4b65-e0d3-49eb-8680-fb8e2ca0296c None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:12.246109 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db7c4b65-e0d3-49eb-8680-fb8e2ca0296c None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:12.246373 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db7c4b65-e0d3-49eb-8680-fb8e2ca0296c None admin] PATH_INFO: `/v3/services` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:12.247022 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db7c4b65-e0d3-49eb-8680-fb8e2ca0296c None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:49:12.247315 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db7c4b65-e0d3-49eb-8680-fb8e2ca0296c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:12.248657 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db7c4b65-e0d3-49eb-8680-fb8e2ca0296c None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:12.258817 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 381/751] 10.222.0.36 () {56 vars in 1122 bytes} [Thu Oct 20 08:49:12 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:49:12.260284 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17d01690-8e16-4871-b6b3-d1c539c4b256 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:12.275829 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:12.368527 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17d01690-8e16-4871-b6b3-d1c539c4b256 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:12.409041 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84a36374-2d1a-4d0b-99f3-16b775fb597e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:12.409652 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84a36374-2d1a-4d0b-99f3-16b775fb597e None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:12.409927 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84a36374-2d1a-4d0b-99f3-16b775fb597e None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:12.410195 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84a36374-2d1a-4d0b-99f3-16b775fb597e None admin] PATH_INFO: `/v3/services` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:12.410843 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84a36374-2d1a-4d0b-99f3-16b775fb597e None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:49:12.411748 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84a36374-2d1a-4d0b-99f3-16b775fb597e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:12.413107 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84a36374-2d1a-4d0b-99f3-16b775fb597e None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:12.418312 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 371/752] 10.222.0.36 () {56 vars in 1122 bytes} [Thu Oct 20 08:49:12 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:49:12.423971 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db7c4b65-e0d3-49eb-8680-fb8e2ca0296c None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:12.490151 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:12.525713 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db7c4b65-e0d3-49eb-8680-fb8e2ca0296c None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:12.560103 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05a1c90a-fd13-4a07-b0f0-44d7b36fcf5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:12.560696 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a1c90a-fd13-4a07-b0f0-44d7b36fcf5e None admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:12.560970 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a1c90a-fd13-4a07-b0f0-44d7b36fcf5e None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:12.561239 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a1c90a-fd13-4a07-b0f0-44d7b36fcf5e None admin] PATH_INFO: `/v3/endpoints` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:12.561909 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05a1c90a-fd13-4a07-b0f0-44d7b36fcf5e None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:12.563288 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05a1c90a-fd13-4a07-b0f0-44d7b36fcf5e None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:12.581224 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 382/753] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 08:49:12 2022] POST /identity/v3/endpoints => generated 325 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 08:49:13.150349 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4acb10bd-d7b7-4f8a-b28e-1aa3135adb01 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:13.151398 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4acb10bd-d7b7-4f8a-b28e-1aa3135adb01 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:13.151913 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4acb10bd-d7b7-4f8a-b28e-1aa3135adb01 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:13.153711 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 372/754] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:49:13 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:49:13.159179 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a75cc0d1-5343-4649-8465-be4eb7928cd1 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:13.160015 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a75cc0d1-5343-4649-8465-be4eb7928cd1 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:13.160490 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a75cc0d1-5343-4649-8465-be4eb7928cd1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:13.180999 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-a75cc0d1-5343-4649-8465-be4eb7928cd1 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:49:13.182225 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-a75cc0d1-5343-4649-8465-be4eb7928cd1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:49:13.214403 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56bfebb-67bf-49d1-a184-108135780197 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:13.214732 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56bfebb-67bf-49d1-a184-108135780197 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:13.214987 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56bfebb-67bf-49d1-a184-108135780197 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:13.216569 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 373/755] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:49:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:49:13.221400 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e9b22a8-747f-4541-9547-b9ea9d38ea37 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:13.221706 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e9b22a8-747f-4541-9547-b9ea9d38ea37 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:13.221982 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e9b22a8-747f-4541-9547-b9ea9d38ea37 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:13.241863 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-7e9b22a8-747f-4541-9547-b9ea9d38ea37 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:49:13.243096 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-7e9b22a8-747f-4541-9547-b9ea9d38ea37 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:49:13.266993 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 383/756] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:49:13 2022] POST /identity/v3/auth/tokens => generated 1813 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 20 08:49:13.292039 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c83b01a7-4d3e-48b8-8658-a15d5f367e32 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:13.292562 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c83b01a7-4d3e-48b8-8658-a15d5f367e32 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:13.293082 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c83b01a7-4d3e-48b8-8658-a15d5f367e32 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:13.294927 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 384/757] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:49:13 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:49:13.299997 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfc260cc-c033-43a6-8d6f-592a884dd27c None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:13.300471 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfc260cc-c033-43a6-8d6f-592a884dd27c None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:13.300963 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfc260cc-c033-43a6-8d6f-592a884dd27c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:13.320851 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-dfc260cc-c033-43a6-8d6f-592a884dd27c None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:49:13.322344 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-dfc260cc-c033-43a6-8d6f-592a884dd27c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:49:13.323711 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 374/758] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:49:13 2022] POST /identity/v3/auth/tokens => generated 1813 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 20 08:49:13.371593 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:13.394535 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 385/759] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:49:13 2022] POST /identity/v3/auth/tokens => generated 1813 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 20 08:49:13.557092 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e9b22a8-747f-4541-9547-b9ea9d38ea37 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:13.560697 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:13.609711 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfc260cc-c033-43a6-8d6f-592a884dd27c None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:13.665635 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e9b22a8-747f-4541-9547-b9ea9d38ea37 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:13.700328 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a91827b5-361d-48b2-a77c-e6d022687774 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:13.700933 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a91827b5-361d-48b2-a77c-e6d022687774 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:13.701233 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a91827b5-361d-48b2-a77c-e6d022687774 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:13.701514 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a91827b5-361d-48b2-a77c-e6d022687774 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:13.702022 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a91827b5-361d-48b2-a77c-e6d022687774 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:13.703623 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a91827b5-361d-48b2-a77c-e6d022687774 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:13.712920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-a91827b5-361d-48b2-a77c-e6d022687774 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Oct 20 08:49:13.712920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:49:13.712920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 20 08:49:13.712920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 20 08:49:13.712920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 20 08:49:13.712920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 20 08:49:13.712920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 20 08:49:13.712920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 20 08:49:13.712920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 20 08:49:13.712920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:49:13.712920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 20 08:49:13.712920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:49:13.712920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:49:13.712920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:49:13.712920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:49:13.712920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:49:13.712920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:49:13.713896 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:49:13.713896 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:49:13.713896 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:49:13.713896 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:49:13.713896 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:49:13.713896 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:49:13.713896 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 20 08:49:13.713896 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 20 08:49:13.713896 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 20 08:49:13.713896 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 20 08:49:13.713896 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:49:13.713896 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:49:13.713896 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:49:13.713896 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:49:13.713896 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:49:13.713896 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:49:13.719787 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:49:13.719787 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:49:13.719787 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:49:13.719787 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:49:13.719787 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:49:13.719787 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:49:13.719787 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:49:13.719787 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:49:13.719787 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:49:13.719787 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:49:13.719787 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 20 08:49:13.719787 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 20 08:49:13.719787 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 20 08:49:13.719787 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:49:13.720732 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfc260cc-c033-43a6-8d6f-592a884dd27c None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:13.720825 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 375/760] 10.222.0.36 () {56 vars in 1111 bytes} [Thu Oct 20 08:49:13 2022] GET /identity/v3/services/compute => generated 88 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:49:13.720825 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a91827b5-361d-48b2-a77c-e6d022687774 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:13.754680 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-676be45d-5708-41af-a662-3cf3b8614133 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:13.755357 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676be45d-5708-41af-a662-3cf3b8614133 None admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:13.755631 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676be45d-5708-41af-a662-3cf3b8614133 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:13.755897 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676be45d-5708-41af-a662-3cf3b8614133 None admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:13.756523 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-676be45d-5708-41af-a662-3cf3b8614133 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:13.757935 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-676be45d-5708-41af-a662-3cf3b8614133 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:13.773063 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.sql.core [None req-676be45d-5708-41af-a662-3cf3b8614133 None admin] Conflict user: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Oct 20 08:49:13.773063 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Oct 20 08:49:13.773063 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [parameters: {'user_id': '3e76925a4f5d462a8531a1df6f04de84', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Oct 20 08:49:13.773063 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81562) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 20 08:49:13.780682 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-676be45d-5708-41af-a662-3cf3b8614133 None admin] Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default.: keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Oct 20 08:49:13.780682 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:49:13.780682 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 20 08:49:13.780682 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 20 08:49:13.780682 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 20 08:49:13.780682 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 20 08:49:13.780682 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 20 08:49:13.780682 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application result = self._query(query) Oct 20 08:49:13.780682 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 20 08:49:13.780682 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application conn.query(q) Oct 20 08:49:13.780682 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 20 08:49:13.780682 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 20 08:49:13.780682 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 20 08:49:13.780682 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application result.read() Oct 20 08:49:13.780682 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 20 08:49:13.781601 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 20 08:49:13.781601 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 20 08:49:13.781601 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 20 08:49:13.781601 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 20 08:49:13.781601 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 20 08:49:13.781601 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 20 08:49:13.781601 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 20 08:49:13.781601 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Oct 20 08:49:13.781601 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:49:13.781601 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 20 08:49:13.781601 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:49:13.781601 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:49:13.781601 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 20 08:49:13.781601 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 20 08:49:13.781601 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 141, in create_user Oct 20 08:49:13.781601 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return base.filter_user(user_ref.to_dict()) Oct 20 08:49:13.781601 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 20 08:49:13.781601 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application next(self.gen) Oct 20 08:49:13.782980 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Oct 20 08:49:13.782980 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application yield resource Oct 20 08:49:13.782980 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 20 08:49:13.782980 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application next(self.gen) Oct 20 08:49:13.782980 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Oct 20 08:49:13.782980 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application self.session.rollback() Oct 20 08:49:13.782980 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 20 08:49:13.782980 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application self.force_reraise() Oct 20 08:49:13.782980 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 20 08:49:13.782980 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise self.value Oct 20 08:49:13.782980 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Oct 20 08:49:13.782980 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 20 08:49:13.782980 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Oct 20 08:49:13.782980 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application session.commit() Oct 20 08:49:13.782980 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 20 08:49:13.782980 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 20 08:49:13.782980 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 20 08:49:13.784018 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application self._prepare_impl() Oct 20 08:49:13.784018 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 20 08:49:13.784018 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application self.session.flush() Oct 20 08:49:13.784018 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 20 08:49:13.784018 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application self._flush(objects) Oct 20 08:49:13.784018 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 20 08:49:13.784018 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 20 08:49:13.784018 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 20 08:49:13.784018 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application compat.raise_( Oct 20 08:49:13.784018 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 20 08:49:13.784018 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception Oct 20 08:49:13.784018 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 20 08:49:13.784018 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application flush_context.execute() Oct 20 08:49:13.784018 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 20 08:49:13.784018 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rec.execute(self) Oct 20 08:49:13.784018 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 20 08:49:13.784018 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 20 08:49:13.784918 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 20 08:49:13.784918 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 20 08:49:13.784918 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements Oct 20 08:49:13.784918 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application result = connection._execute_20( Oct 20 08:49:13.784918 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 20 08:49:13.784918 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 20 08:49:13.784918 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 20 08:49:13.784918 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 20 08:49:13.784918 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 20 08:49:13.784918 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 20 08:49:13.784918 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 20 08:49:13.784918 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 20 08:49:13.784918 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 20 08:49:13.784918 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 20 08:49:13.784918 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 20 08:49:13.784918 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception Oct 20 08:49:13.785768 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 20 08:49:13.785768 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 20 08:49:13.785768 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 20 08:49:13.785768 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 20 08:49:13.785768 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 20 08:49:13.785768 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application result = self._query(query) Oct 20 08:49:13.785768 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 20 08:49:13.785768 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application conn.query(q) Oct 20 08:49:13.785768 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 20 08:49:13.785768 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 20 08:49:13.785768 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 20 08:49:13.785768 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application result.read() Oct 20 08:49:13.785768 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 20 08:49:13.785768 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 20 08:49:13.785768 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 20 08:49:13.785768 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 20 08:49:13.789842 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 20 08:49:13.789842 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 20 08:49:13.789842 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 20 08:49:13.789842 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 20 08:49:13.789842 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Oct 20 08:49:13.789842 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Oct 20 08:49:13.789842 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [parameters: {'user_id': '3e76925a4f5d462a8531a1df6f04de84', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Oct 20 08:49:13.789842 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 20 08:49:13.789842 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:49:13.789842 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 20 08:49:13.789842 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:49:13.789842 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:49:13.789842 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:49:13.789842 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:49:13.789842 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:49:13.793188 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:49:13.793188 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:49:13.793188 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:49:13.793188 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:49:13.793188 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:49:13.793188 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:49:13.793188 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:49:13.793188 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 244, in post Oct 20 08:49:13.793188 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.create_user( Oct 20 08:49:13.793188 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:49:13.793188 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:49:13.793188 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 20 08:49:13.793188 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 20 08:49:13.793188 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 20 08:49:13.793188 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 20 08:49:13.793188 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1000, in create_user Oct 20 08:49:13.793188 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application ref = self._create_user_with_federated_objects(user, driver) Oct 20 08:49:13.797049 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 966, in _create_user_with_federated_objects Oct 20 08:49:13.797049 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application user = driver.create_user(user['id'], user) Oct 20 08:49:13.797049 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 20 08:49:13.797049 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 20 08:49:13.797049 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Oct 20 08:49:13.797049 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:49:13.797049 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 386/761] 10.222.0.36 () {60 vars in 1131 bytes} [Thu Oct 20 08:49:13 2022] POST /identity/v3/users => generated 160 bytes in 173 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Oct 20 08:49:13.797049 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-676be45d-5708-41af-a662-3cf3b8614133 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:13.797890 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:13.826358 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a91827b5-361d-48b2-a77c-e6d022687774 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:13.838094 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:13.862828 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e50e0651-4a3c-42cd-9f2d-a6baef11716b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:13.863590 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50e0651-4a3c-42cd-9f2d-a6baef11716b None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:13.863941 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50e0651-4a3c-42cd-9f2d-a6baef11716b None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:13.864805 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50e0651-4a3c-42cd-9f2d-a6baef11716b None admin] PATH_INFO: `/v3/services/image` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:13.865454 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e50e0651-4a3c-42cd-9f2d-a6baef11716b None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:13.866954 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e50e0651-4a3c-42cd-9f2d-a6baef11716b None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:13.875773 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-e50e0651-4a3c-42cd-9f2d-a6baef11716b None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Oct 20 08:49:13.875773 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:49:13.875773 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 20 08:49:13.875773 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 20 08:49:13.875773 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 20 08:49:13.875773 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 20 08:49:13.875773 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 20 08:49:13.875773 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 20 08:49:13.875773 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 20 08:49:13.875773 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:49:13.875773 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 20 08:49:13.875773 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:49:13.875773 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:49:13.875773 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:49:13.875773 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:49:13.875773 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:49:13.875773 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:49:13.877017 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:49:13.877017 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:49:13.877017 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:49:13.877017 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:49:13.877017 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:49:13.877017 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:49:13.877017 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 20 08:49:13.877017 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 20 08:49:13.877017 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 20 08:49:13.877017 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 20 08:49:13.877017 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:49:13.877017 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:49:13.877017 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:49:13.877017 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:49:13.877017 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:49:13.877017 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:49:13.878050 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:49:13.878050 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:49:13.878050 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:49:13.878050 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:49:13.878050 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:49:13.878050 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:49:13.878050 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:49:13.878050 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:49:13.878050 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:49:13.878050 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:49:13.878050 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 20 08:49:13.878050 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 20 08:49:13.878050 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 20 08:49:13.878050 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:49:13.884483 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 376/762] 10.222.0.36 () {56 vars in 1105 bytes} [Thu Oct 20 08:49:13 2022] GET /identity/v3/services/image => generated 86 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:49:13.884483 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e50e0651-4a3c-42cd-9f2d-a6baef11716b None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:13.906717 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-676be45d-5708-41af-a662-3cf3b8614133 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:13.942924 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7da4b94b-f997-4137-8f71-233719dcbdbb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:13.943532 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7da4b94b-f997-4137-8f71-233719dcbdbb None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:13.943802 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7da4b94b-f997-4137-8f71-233719dcbdbb None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:13.944058 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7da4b94b-f997-4137-8f71-233719dcbdbb None admin] PATH_INFO: `/v3/services` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:13.944687 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7da4b94b-f997-4137-8f71-233719dcbdbb None admin] RBAC: Adding query filter params (name=compute) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:49:13.944977 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7da4b94b-f997-4137-8f71-233719dcbdbb None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:13.946549 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7da4b94b-f997-4137-8f71-233719dcbdbb None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:13.955812 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 387/763] 10.222.0.36 () {56 vars in 1112 bytes} [Thu Oct 20 08:49:13 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:49:13.956601 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7da4b94b-f997-4137-8f71-233719dcbdbb None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:13.999780 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e50e0651-4a3c-42cd-9f2d-a6baef11716b None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:14.035457 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:14.042287 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df83246e-8001-40bb-a2e2-63424e54a7bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:14.043050 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df83246e-8001-40bb-a2e2-63424e54a7bb None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:14.043325 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df83246e-8001-40bb-a2e2-63424e54a7bb None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:14.043735 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df83246e-8001-40bb-a2e2-63424e54a7bb None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:14.052538 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df83246e-8001-40bb-a2e2-63424e54a7bb None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:14.054088 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df83246e-8001-40bb-a2e2-63424e54a7bb None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:14.058195 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7da4b94b-f997-4137-8f71-233719dcbdbb None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:14.066161 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-df83246e-8001-40bb-a2e2-63424e54a7bb None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 20 08:49:14.066161 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:49:14.066161 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:49:14.066161 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:49:14.066161 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:49:14.066161 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:49:14.066161 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:49:14.066161 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:49:14.066161 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:49:14.066161 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:49:14.066161 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:49:14.066161 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:49:14.066161 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 20 08:49:14.066161 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 20 08:49:14.066161 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 20 08:49:14.066161 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 20 08:49:14.067152 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:49:14.067152 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:49:14.067152 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 20 08:49:14.067152 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 20 08:49:14.067152 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 20 08:49:14.067152 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 20 08:49:14.067152 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:49:14.067152 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:49:14.067152 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:49:14.067152 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:49:14.067152 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:49:14.067152 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:49:14.067152 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:49:14.067152 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:49:14.067152 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:49:14.067152 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:49:14.067152 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:49:14.067152 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:49:14.073314 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:49:14.073314 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:49:14.073314 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 20 08:49:14.073314 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 20 08:49:14.073314 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 20 08:49:14.073314 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 20 08:49:14.073314 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 20 08:49:14.073314 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 20 08:49:14.073314 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 20 08:49:14.073314 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:49:14.073957 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:14.074032 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 377/764] 10.222.0.36 () {56 vars in 1099 bytes} [Thu Oct 20 08:49:13 2022] GET /identity/v3/users/cinder => generated 84 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:49:14.074032 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df83246e-8001-40bb-a2e2-63424e54a7bb None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:14.101250 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f7baa44-9357-400b-9d0f-d9ce662480e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:14.101863 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7baa44-9357-400b-9d0f-d9ce662480e6 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:14.102202 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7baa44-9357-400b-9d0f-d9ce662480e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:14.102565 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7baa44-9357-400b-9d0f-d9ce662480e6 None admin] PATH_INFO: `/v3/services` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:14.103249 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f7baa44-9357-400b-9d0f-d9ce662480e6 None admin] RBAC: Adding query filter params (name=image) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:49:14.103624 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f7baa44-9357-400b-9d0f-d9ce662480e6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:14.105045 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f7baa44-9357-400b-9d0f-d9ce662480e6 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:14.114714 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 388/765] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:49:13 2022] GET /identity/v3/services?name=image => generated 122 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:49:14.114714 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f7baa44-9357-400b-9d0f-d9ce662480e6 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:14.183755 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df83246e-8001-40bb-a2e2-63424e54a7bb None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:14.211580 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f7baa44-9357-400b-9d0f-d9ce662480e6 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:14.219812 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bc220da-1278-4121-8b88-10bbbf46cefc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:14.220702 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc220da-1278-4121-8b88-10bbbf46cefc None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:14.221209 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc220da-1278-4121-8b88-10bbbf46cefc None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:14.221781 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc220da-1278-4121-8b88-10bbbf46cefc None admin] PATH_INFO: `/v3/services` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:14.222417 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bc220da-1278-4121-8b88-10bbbf46cefc None admin] RBAC: Adding query filter params (type=compute) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:49:14.222748 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bc220da-1278-4121-8b88-10bbbf46cefc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:14.224691 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bc220da-1278-4121-8b88-10bbbf46cefc None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:14.234574 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 378/766] 10.222.0.36 () {56 vars in 1112 bytes} [Thu Oct 20 08:49:14 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:49:14.234574 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bc220da-1278-4121-8b88-10bbbf46cefc None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:14.249401 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e770b7ef-3980-489c-a53a-7c584931adb1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:14.249997 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e770b7ef-3980-489c-a53a-7c584931adb1 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:14.250272 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e770b7ef-3980-489c-a53a-7c584931adb1 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:14.250580 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e770b7ef-3980-489c-a53a-7c584931adb1 None admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:14.251270 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e770b7ef-3980-489c-a53a-7c584931adb1 None admin] RBAC: Adding query filter params (name=cinder) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:49:14.251630 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e770b7ef-3980-489c-a53a-7c584931adb1 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:14.253030 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e770b7ef-3980-489c-a53a-7c584931adb1 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:14.288098 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 389/767] 10.222.0.36 () {56 vars in 1101 bytes} [Thu Oct 20 08:49:14 2022] GET /identity/v3/users?name=cinder => generated 354 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:49:14.288926 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e770b7ef-3980-489c-a53a-7c584931adb1 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:14.304415 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:14.342686 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:14.363085 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bc220da-1278-4121-8b88-10bbbf46cefc None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:14.403826 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2654e3a-9f24-40d1-95f2-688bfcbb69e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:14.404834 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2654e3a-9f24-40d1-95f2-688bfcbb69e3 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:14.405304 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2654e3a-9f24-40d1-95f2-688bfcbb69e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:14.405611 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2654e3a-9f24-40d1-95f2-688bfcbb69e3 None admin] PATH_INFO: `/v3/services` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:14.406236 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2654e3a-9f24-40d1-95f2-688bfcbb69e3 None admin] RBAC: Adding query filter params (type=image) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:49:14.406609 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2654e3a-9f24-40d1-95f2-688bfcbb69e3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:14.408032 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2654e3a-9f24-40d1-95f2-688bfcbb69e3 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:14.414629 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 379/768] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:49:14 2022] GET /identity/v3/services?type=image => generated 350 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:49:14.420840 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2654e3a-9f24-40d1-95f2-688bfcbb69e3 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:14.454554 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e770b7ef-3980-489c-a53a-7c584931adb1 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:14.502549 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e835de-0af6-459c-bb3a-f67f52a1ca78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:14.503186 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e835de-0af6-459c-bb3a-f67f52a1ca78 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:14.503461 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e835de-0af6-459c-bb3a-f67f52a1ca78 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:14.503731 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e835de-0af6-459c-bb3a-f67f52a1ca78 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:14.504422 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54e835de-0af6-459c-bb3a-f67f52a1ca78 None admin] RBAC: Adding query filter params (interface=public, service_id=19a2fac060094816828ab687b8c68a6e, region_id=RegionOne) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:49:14.504787 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54e835de-0af6-459c-bb3a-f67f52a1ca78 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:14.506706 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54e835de-0af6-459c-bb3a-f67f52a1ca78 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:14.515233 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 390/769] 10.222.0.36 () {56 vars in 1251 bytes} [Thu Oct 20 08:49:14 2022] GET /identity/v3/endpoints?service_id=19a2fac060094816828ab687b8c68a6e&interface=public®ion_id=RegionOne => generated 194 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:49:14.522794 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e835de-0af6-459c-bb3a-f67f52a1ca78 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:14.637076 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:14.644543 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e835de-0af6-459c-bb3a-f67f52a1ca78 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:14.662865 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2654e3a-9f24-40d1-95f2-688bfcbb69e3 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:14.681207 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99ce8589-f591-4a34-9d64-ec1b8a55bb74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:14.681803 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ce8589-f591-4a34-9d64-ec1b8a55bb74 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:14.682077 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ce8589-f591-4a34-9d64-ec1b8a55bb74 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:14.682344 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ce8589-f591-4a34-9d64-ec1b8a55bb74 None admin] PATH_INFO: `/v3/services` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:14.682939 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:14.683058 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99ce8589-f591-4a34-9d64-ec1b8a55bb74 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:14.684329 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99ce8589-f591-4a34-9d64-ec1b8a55bb74 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:14.691333 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 391/770] 10.222.0.36 () {56 vars in 1087 bytes} [Thu Oct 20 08:49:14 2022] GET /identity/v3/services => generated 1957 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 08:49:14.699933 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55bff112-1784-4b3b-b672-a1f5222e3e7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:14.700580 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55bff112-1784-4b3b-b672-a1f5222e3e7e None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:14.700859 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55bff112-1784-4b3b-b672-a1f5222e3e7e None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:14.701130 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55bff112-1784-4b3b-b672-a1f5222e3e7e None admin] PATH_INFO: `/v3/endpoints` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:14.701879 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55bff112-1784-4b3b-b672-a1f5222e3e7e None admin] RBAC: Adding query filter params (interface=public, service_id=b6ef181903e541f993bc93a3aebda0f5, region_id=RegionOne) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:49:14.702193 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55bff112-1784-4b3b-b672-a1f5222e3e7e None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:14.703743 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55bff112-1784-4b3b-b672-a1f5222e3e7e None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:14.711214 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 380/771] 10.222.0.36 () {56 vars in 1251 bytes} [Thu Oct 20 08:49:14 2022] GET /identity/v3/endpoints?service_id=b6ef181903e541f993bc93a3aebda0f5&interface=public®ion_id=RegionOne => generated 194 bytes in 291 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 08:49:14.717226 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99ce8589-f591-4a34-9d64-ec1b8a55bb74 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:14.823684 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0daccd8b-5f6e-4cd0-aab8-f54b600e12b5 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:14.824017 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0daccd8b-5f6e-4cd0-aab8-f54b600e12b5 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:14.824231 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0daccd8b-5f6e-4cd0-aab8-f54b600e12b5 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:14.825609 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 381/772] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:49:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 20 08:49:14.830043 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99ce8589-f591-4a34-9d64-ec1b8a55bb74 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:14.831320 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-170eeca0-ad5e-47ab-9c96-750df0995765 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:14.831611 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-170eeca0-ad5e-47ab-9c96-750df0995765 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:14.831881 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-170eeca0-ad5e-47ab-9c96-750df0995765 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:14.851279 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-170eeca0-ad5e-47ab-9c96-750df0995765 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:49:14.852531 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-170eeca0-ad5e-47ab-9c96-750df0995765 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:49:14.869085 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca66300f-d972-426f-9387-2c561d0f6c40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:14.869688 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca66300f-d972-426f-9387-2c561d0f6c40 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:14.869961 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca66300f-d972-426f-9387-2c561d0f6c40 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:14.870227 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca66300f-d972-426f-9387-2c561d0f6c40 None admin] PATH_INFO: `/v3/services` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:14.870859 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca66300f-d972-426f-9387-2c561d0f6c40 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:14.872207 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca66300f-d972-426f-9387-2c561d0f6c40 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:14.879636 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 392/773] 10.222.0.36 () {56 vars in 1087 bytes} [Thu Oct 20 08:49:14 2022] GET /identity/v3/services => generated 1957 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 08:49:14.932414 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 382/774] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:49:14 2022] POST /identity/v3/auth/tokens => generated 1813 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 20 08:49:15.200671 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca66300f-d972-426f-9387-2c561d0f6c40 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:15.226523 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:15.301905 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca66300f-d972-426f-9387-2c561d0f6c40 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:15.336043 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c3c4808-5e78-4931-b039-002d2fe316ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:15.336669 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c3c4808-5e78-4931-b039-002d2fe316ff None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:15.336941 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c3c4808-5e78-4931-b039-002d2fe316ff None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:15.337202 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c3c4808-5e78-4931-b039-002d2fe316ff None admin] PATH_INFO: `/v3/domains/default` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:15.338282 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c3c4808-5e78-4931-b039-002d2fe316ff None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:15.339728 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c3c4808-5e78-4931-b039-002d2fe316ff None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:15.341434 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 393/775] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:49:15 2022] GET /identity/v3/domains/default => generated 199 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:49:15.347133 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-170eeca0-ad5e-47ab-9c96-750df0995765 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:15.456309 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-170eeca0-ad5e-47ab-9c96-750df0995765 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:15.471304 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:15.492203 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e60a1018-65b9-4ece-aa96-b9e1412ac4f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:15.492804 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e60a1018-65b9-4ece-aa96-b9e1412ac4f8 None admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:15.493075 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e60a1018-65b9-4ece-aa96-b9e1412ac4f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:15.493337 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e60a1018-65b9-4ece-aa96-b9e1412ac4f8 None admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:15.493986 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e60a1018-65b9-4ece-aa96-b9e1412ac4f8 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:15.495433 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e60a1018-65b9-4ece-aa96-b9e1412ac4f8 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:15.523948 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 383/776] 10.222.0.36 () {60 vars in 1141 bytes} [Thu Oct 20 08:49:15 2022] POST /identity/v3/projects => generated 311 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 08:49:16.875744 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e14476f-dff5-4970-986c-596d46cede59 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:16.876287 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e14476f-dff5-4970-986c-596d46cede59 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:16.876287 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e14476f-dff5-4970-986c-596d46cede59 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:16.877747 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 394/777] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:49:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:49:16.883592 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-835dedeb-57fd-444d-943c-7d9e06a05178 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:16.883987 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-835dedeb-57fd-444d-943c-7d9e06a05178 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:16.884170 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-835dedeb-57fd-444d-943c-7d9e06a05178 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:16.901896 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-835dedeb-57fd-444d-943c-7d9e06a05178 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:49:16.903099 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-835dedeb-57fd-444d-943c-7d9e06a05178 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:49:16.974803 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 384/778] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:49:16 2022] POST /identity/v3/auth/tokens => generated 1813 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 20 08:49:17.034841 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cebd5340-2f06-4352-a139-a82af1db4bdd None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:17.035124 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cebd5340-2f06-4352-a139-a82af1db4bdd None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:17.035398 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cebd5340-2f06-4352-a139-a82af1db4bdd None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:17.037262 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 395/779] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:49:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:49:17.042668 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f39019-0527-4222-a83e-e8768e5804f2 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:17.042976 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f39019-0527-4222-a83e-e8768e5804f2 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:17.043241 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47f39019-0527-4222-a83e-e8768e5804f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:17.061655 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-47f39019-0527-4222-a83e-e8768e5804f2 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:49:17.062849 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-47f39019-0527-4222-a83e-e8768e5804f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:49:17.131813 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 385/780] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:49:17 2022] POST /identity/v3/auth/tokens => generated 1813 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 20 08:49:17.244371 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cebd5340-2f06-4352-a139-a82af1db4bdd None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:17.333907 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:17.345559 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cebd5340-2f06-4352-a139-a82af1db4bdd None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:17.386147 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-072d2794-6812-4825-818d-38e5162cd1f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:17.386746 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-072d2794-6812-4825-818d-38e5162cd1f7 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:17.387025 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-072d2794-6812-4825-818d-38e5162cd1f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:17.387300 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-072d2794-6812-4825-818d-38e5162cd1f7 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:17.387880 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-072d2794-6812-4825-818d-38e5162cd1f7 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:17.389298 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-072d2794-6812-4825-818d-38e5162cd1f7 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:17.398802 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-072d2794-6812-4825-818d-38e5162cd1f7 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Oct 20 08:49:17.398802 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:49:17.398802 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 20 08:49:17.398802 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 20 08:49:17.398802 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 20 08:49:17.398802 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 20 08:49:17.398802 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 20 08:49:17.398802 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 20 08:49:17.398802 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 20 08:49:17.398802 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:49:17.398802 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 20 08:49:17.398802 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:49:17.398802 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:49:17.398802 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:49:17.398802 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:49:17.398802 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:49:17.398802 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:49:17.399775 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:49:17.399775 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:49:17.399775 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:49:17.399775 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:49:17.399775 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:49:17.399775 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:49:17.399775 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 20 08:49:17.399775 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 20 08:49:17.399775 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 20 08:49:17.399775 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 20 08:49:17.399775 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:49:17.399775 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:49:17.399775 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:49:17.399775 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:49:17.399775 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:49:17.399775 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:49:17.400620 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:49:17.400620 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:49:17.400620 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:49:17.400620 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:49:17.400620 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:49:17.400620 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:49:17.400620 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:49:17.400620 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:49:17.400620 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:49:17.400620 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:49:17.400620 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 20 08:49:17.400620 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 20 08:49:17.400620 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 20 08:49:17.400620 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:49:17.400620 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 396/781] 10.222.0.36 () {56 vars in 1111 bytes} [Thu Oct 20 08:49:17 2022] GET /identity/v3/services/compute => generated 88 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:49:17.404586 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f39019-0527-4222-a83e-e8768e5804f2 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:17.407310 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-072d2794-6812-4825-818d-38e5162cd1f7 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:17.433302 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:17.509620 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47f39019-0527-4222-a83e-e8768e5804f2 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:17.524251 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-072d2794-6812-4825-818d-38e5162cd1f7 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:17.544862 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9822e825-9981-4c3d-b528-01bd45ab839e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:17.545485 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9822e825-9981-4c3d-b528-01bd45ab839e None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:17.545760 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9822e825-9981-4c3d-b528-01bd45ab839e None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:17.546022 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9822e825-9981-4c3d-b528-01bd45ab839e None admin] PATH_INFO: `/v3/services/image` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:17.546625 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9822e825-9981-4c3d-b528-01bd45ab839e None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:17.547986 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9822e825-9981-4c3d-b528-01bd45ab839e None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:17.556410 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-9822e825-9981-4c3d-b528-01bd45ab839e None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Oct 20 08:49:17.556410 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:49:17.556410 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 20 08:49:17.556410 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 20 08:49:17.556410 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 20 08:49:17.556410 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 20 08:49:17.556410 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 20 08:49:17.556410 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 20 08:49:17.556410 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 20 08:49:17.556410 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:49:17.556410 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 20 08:49:17.556410 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:49:17.556410 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:49:17.556410 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:49:17.556410 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:49:17.556410 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:49:17.556410 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:49:17.557425 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:49:17.557425 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:49:17.557425 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:49:17.557425 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:49:17.557425 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:49:17.557425 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:49:17.557425 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 20 08:49:17.557425 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 20 08:49:17.557425 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 20 08:49:17.557425 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 20 08:49:17.557425 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:49:17.557425 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:49:17.557425 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:49:17.557425 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:49:17.557425 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:49:17.557425 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:49:17.558287 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:49:17.558287 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:49:17.558287 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:49:17.558287 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:49:17.558287 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:49:17.558287 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:49:17.558287 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:49:17.558287 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:49:17.558287 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:49:17.558287 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:49:17.558287 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 20 08:49:17.558287 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 20 08:49:17.558287 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 20 08:49:17.558287 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:49:17.558287 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 386/782] 10.222.0.36 () {56 vars in 1105 bytes} [Thu Oct 20 08:49:17 2022] GET /identity/v3/services/image => generated 86 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:49:17.563228 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9822e825-9981-4c3d-b528-01bd45ab839e None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:17.568139 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-469abf95-de37-4a7d-8e2a-e41494634691 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:17.568940 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-469abf95-de37-4a7d-8e2a-e41494634691 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:17.569371 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-469abf95-de37-4a7d-8e2a-e41494634691 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:17.569792 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-469abf95-de37-4a7d-8e2a-e41494634691 None admin] PATH_INFO: `/v3/services` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:17.570609 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-469abf95-de37-4a7d-8e2a-e41494634691 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:49:17.571076 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-469abf95-de37-4a7d-8e2a-e41494634691 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:17.573121 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-469abf95-de37-4a7d-8e2a-e41494634691 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:17.579720 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 397/783] 10.222.0.36 () {56 vars in 1112 bytes} [Thu Oct 20 08:49:17 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:49:17.586098 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-469abf95-de37-4a7d-8e2a-e41494634691 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:17.608923 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:17.678877 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:17.690679 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9822e825-9981-4c3d-b528-01bd45ab839e None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:17.694955 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-469abf95-de37-4a7d-8e2a-e41494634691 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:17.725062 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39f86391-3ebb-40e0-833c-0cf57717c781 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:17.725668 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f86391-3ebb-40e0-833c-0cf57717c781 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:17.725947 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f86391-3ebb-40e0-833c-0cf57717c781 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:17.726216 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f86391-3ebb-40e0-833c-0cf57717c781 None admin] PATH_INFO: `/v3/services` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:17.726907 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39f86391-3ebb-40e0-833c-0cf57717c781 None admin] RBAC: Adding query filter params (name=image) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:49:17.727212 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39f86391-3ebb-40e0-833c-0cf57717c781 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:17.728579 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39f86391-3ebb-40e0-833c-0cf57717c781 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:17.730765 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff24f04-21d5-47ee-949e-4ef12f4dba81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:17.731390 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff24f04-21d5-47ee-949e-4ef12f4dba81 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:17.731673 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff24f04-21d5-47ee-949e-4ef12f4dba81 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:17.731946 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff24f04-21d5-47ee-949e-4ef12f4dba81 None admin] PATH_INFO: `/v3/services` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:17.732565 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ff24f04-21d5-47ee-949e-4ef12f4dba81 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:49:17.732867 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ff24f04-21d5-47ee-949e-4ef12f4dba81 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:17.733388 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 387/784] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:49:17 2022] GET /identity/v3/services?name=image => generated 122 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 08:49:17.734253 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ff24f04-21d5-47ee-949e-4ef12f4dba81 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:17.734541 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a316d325-a2f6-437a-a32a-27278985f0ef None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:17.734827 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a316d325-a2f6-437a-a32a-27278985f0ef None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:17.735094 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a316d325-a2f6-437a-a32a-27278985f0ef None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:17.736279 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 388/785] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:49:17 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:49:17.741004 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a316d325-a2f6-437a-a32a-27278985f0ef None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:17.742067 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 398/786] 10.222.0.36 () {56 vars in 1112 bytes} [Thu Oct 20 08:49:17 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:49:17.743170 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1696f09-e5f0-40b4-8d18-3fcb15e610f1 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:17.743459 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1696f09-e5f0-40b4-8d18-3fcb15e610f1 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:17.743734 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1696f09-e5f0-40b4-8d18-3fcb15e610f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:17.761791 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-f1696f09-e5f0-40b4-8d18-3fcb15e610f1 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:49:17.763325 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-f1696f09-e5f0-40b4-8d18-3fcb15e610f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:49:17.828185 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 399/787] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:49:17 2022] POST /identity/v3/auth/tokens => generated 1813 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 20 08:49:17.828973 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1696f09-e5f0-40b4-8d18-3fcb15e610f1 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:17.843581 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:17.844444 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a316d325-a2f6-437a-a32a-27278985f0ef None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:17.881459 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-547c4583-4005-4771-883c-b4e64e33c004 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:17.882369 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-547c4583-4005-4771-883c-b4e64e33c004 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:17.882674 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-547c4583-4005-4771-883c-b4e64e33c004 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:17.882944 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-547c4583-4005-4771-883c-b4e64e33c004 None admin] PATH_INFO: `/v3/services` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:17.883575 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-547c4583-4005-4771-883c-b4e64e33c004 None admin] RBAC: Adding query filter params (type=image) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:49:17.883874 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-547c4583-4005-4771-883c-b4e64e33c004 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:17.885236 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-547c4583-4005-4771-883c-b4e64e33c004 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:17.889958 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 389/788] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:49:17 2022] GET /identity/v3/services?type=image => generated 350 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:49:17.896102 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-547c4583-4005-4771-883c-b4e64e33c004 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:17.914742 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:17.939693 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1696f09-e5f0-40b4-8d18-3fcb15e610f1 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:17.975244 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a39dd6bc-71d8-4436-bb7b-5ccfd36c59f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:17.975845 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a39dd6bc-71d8-4436-bb7b-5ccfd36c59f8 None admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:17.976122 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a39dd6bc-71d8-4436-bb7b-5ccfd36c59f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:17.976384 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a39dd6bc-71d8-4436-bb7b-5ccfd36c59f8 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:17.977038 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a39dd6bc-71d8-4436-bb7b-5ccfd36c59f8 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:17.978768 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a39dd6bc-71d8-4436-bb7b-5ccfd36c59f8 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:18.005716 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-547c4583-4005-4771-883c-b4e64e33c004 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:18.010676 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 400/789] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 08:49:17 2022] POST /identity/v3/endpoints => generated 334 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 08:49:18.054667 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fefb213-229f-454b-852a-2ffdd961ec90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:18.055708 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fefb213-229f-454b-852a-2ffdd961ec90 None admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:18.056174 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fefb213-229f-454b-852a-2ffdd961ec90 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:18.056680 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fefb213-229f-454b-852a-2ffdd961ec90 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:18.057594 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fefb213-229f-454b-852a-2ffdd961ec90 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:18.059318 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fefb213-229f-454b-852a-2ffdd961ec90 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:18.086511 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 390/790] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 08:49:17 2022] POST /identity/v3/endpoints => generated 327 bytes in 187 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 20 08:49:18.096704 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a39dd6bc-71d8-4436-bb7b-5ccfd36c59f8 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:18.172844 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:18.215406 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a39dd6bc-71d8-4436-bb7b-5ccfd36c59f8 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:18.259256 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecc5b891-67d9-475d-92f1-cec0bded51da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:18.259891 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecc5b891-67d9-475d-92f1-cec0bded51da None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:18.260166 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecc5b891-67d9-475d-92f1-cec0bded51da None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:18.260430 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecc5b891-67d9-475d-92f1-cec0bded51da None admin] PATH_INFO: `/v3/domains/default` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:18.261549 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecc5b891-67d9-475d-92f1-cec0bded51da None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:18.263046 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecc5b891-67d9-475d-92f1-cec0bded51da None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:18.264825 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 401/791] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:49:18 2022] GET /identity/v3/domains/default => generated 199 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:49:18.271229 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fefb213-229f-454b-852a-2ffdd961ec90 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:18.344355 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:18.385049 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fefb213-229f-454b-852a-2ffdd961ec90 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:18.420368 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab643f38-6193-4f7b-8150-ceca8486107f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:18.420983 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab643f38-6193-4f7b-8150-ceca8486107f None admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:18.421256 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab643f38-6193-4f7b-8150-ceca8486107f None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:18.421519 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab643f38-6193-4f7b-8150-ceca8486107f None admin] PATH_INFO: `/v3/users` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:18.422219 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab643f38-6193-4f7b-8150-ceca8486107f None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:18.423686 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab643f38-6193-4f7b-8150-ceca8486107f None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:18.454508 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 391/792] 10.222.0.36 () {60 vars in 1132 bytes} [Thu Oct 20 08:49:18 2022] POST /identity/v3/users => generated 285 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 08:49:20.257814 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd1c2e6-41b8-44bb-b8f6-7fd336958056 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:20.258350 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd1c2e6-41b8-44bb-b8f6-7fd336958056 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:20.258350 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd1c2e6-41b8-44bb-b8f6-7fd336958056 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:20.262519 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 402/793] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:49:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:49:20.265011 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8db652-d4cb-480f-87ae-3382c5776e2c None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:20.265414 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8db652-d4cb-480f-87ae-3382c5776e2c None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:20.265592 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8db652-d4cb-480f-87ae-3382c5776e2c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:20.284718 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-8b8db652-d4cb-480f-87ae-3382c5776e2c None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:49:20.285976 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-8b8db652-d4cb-480f-87ae-3382c5776e2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:49:20.297679 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f455820-c3b2-4a92-b8e9-2c889d615fc1 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:20.297963 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f455820-c3b2-4a92-b8e9-2c889d615fc1 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:20.298233 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f455820-c3b2-4a92-b8e9-2c889d615fc1 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:20.299723 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 403/794] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:49:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 20 08:49:20.304806 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d91fd5-4b27-452f-a1ff-2b73314d5783 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:20.305088 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d91fd5-4b27-452f-a1ff-2b73314d5783 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:20.305357 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d91fd5-4b27-452f-a1ff-2b73314d5783 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:20.324535 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-90d91fd5-4b27-452f-a1ff-2b73314d5783 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:49:20.325716 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-90d91fd5-4b27-452f-a1ff-2b73314d5783 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:49:20.355508 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 392/795] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:49:20 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 20 08:49:20.406515 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 404/796] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:49:20 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 20 08:49:20.606300 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b5badc-f7ee-4a48-a786-563e1b662a88 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:20.606674 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b5badc-f7ee-4a48-a786-563e1b662a88 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:20.606886 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b5badc-f7ee-4a48-a786-563e1b662a88 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:20.608263 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 393/797] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:49:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:49:20.613298 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d32368b1-d801-48c8-96b9-8d8448a79c3f None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:20.613593 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d32368b1-d801-48c8-96b9-8d8448a79c3f None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:20.613876 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d32368b1-d801-48c8-96b9-8d8448a79c3f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:20.626143 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9b5badc-f7ee-4a48-a786-563e1b662a88 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:20.632829 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-d32368b1-d801-48c8-96b9-8d8448a79c3f None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:49:20.634015 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-d32368b1-d801-48c8-96b9-8d8448a79c3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:49:20.684193 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:20.694082 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:20.704152 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 405/798] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:49:20 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 20 08:49:20.705219 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d32368b1-d801-48c8-96b9-8d8448a79c3f None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:20.730368 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9b5badc-f7ee-4a48-a786-563e1b662a88 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:20.766088 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed05c36-ded7-42d0-9a87-0b5e1d854c7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:20.767005 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed05c36-ded7-42d0-9a87-0b5e1d854c7f None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:20.767480 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed05c36-ded7-42d0-9a87-0b5e1d854c7f None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:20.767755 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed05c36-ded7-42d0-9a87-0b5e1d854c7f None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:20.778801 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bed05c36-ded7-42d0-9a87-0b5e1d854c7f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 20 08:49:20.783548 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-bed05c36-ded7-42d0-9a87-0b5e1d854c7f None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 20 08:49:20.783548 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:49:20.783548 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:49:20.783548 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:49:20.783548 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:49:20.783548 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:49:20.783548 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:49:20.783548 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:49:20.783548 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:49:20.783548 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:49:20.783548 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:49:20.783548 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:49:20.783548 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 20 08:49:20.783548 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 20 08:49:20.783548 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 20 08:49:20.783548 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 20 08:49:20.784482 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 20 08:49:20.784482 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 20 08:49:20.784482 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 20 08:49:20.784482 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 20 08:49:20.784482 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:49:20.784482 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:49:20.784482 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:49:20.784482 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:49:20.784482 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:49:20.784482 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:49:20.784482 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:49:20.784482 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:49:20.784482 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:49:20.784482 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:49:20.784482 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:49:20.784482 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:49:20.787442 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:49:20.787442 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:49:20.787442 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:49:20.787442 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:49:20.787442 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 20 08:49:20.787442 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 20 08:49:20.787442 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 20 08:49:20.787442 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 20 08:49:20.787442 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 20 08:49:20.787442 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 20 08:49:20.787442 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 20 08:49:20.787442 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:49:20.787442 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 394/799] 10.222.0.36 () {56 vars in 1120 bytes} [Thu Oct 20 08:49:20 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:49:20.790906 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed05c36-ded7-42d0-9a87-0b5e1d854c7f None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:20.816861 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d32368b1-d801-48c8-96b9-8d8448a79c3f None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:20.853415 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80390d94-8fb4-4bcb-9e9b-936575b13bf6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:20.854010 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80390d94-8fb4-4bcb-9e9b-936575b13bf6 None admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:20.854281 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80390d94-8fb4-4bcb-9e9b-936575b13bf6 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:20.854579 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80390d94-8fb4-4bcb-9e9b-936575b13bf6 None admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:20.855230 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80390d94-8fb4-4bcb-9e9b-936575b13bf6 None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:20.856632 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80390d94-8fb4-4bcb-9e9b-936575b13bf6 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:20.866749 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.sql.core [None req-80390d94-8fb4-4bcb-9e9b-936575b13bf6 None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 20 08:49:20.866749 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 20 08:49:20.866749 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [parameters: {'id': '3062eb2398aa4a5ebaebc95c47bb70a8', 'name': 'Default', 'domain_id': '<>', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': None, 'is_domain': 1}] Oct 20 08:49:20.866749 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81562) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 20 08:49:20.873827 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-80390d94-8fb4-4bcb-9e9b-936575b13bf6 None admin] Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default. Oct 20 08:49:20.873827 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:49:20.873827 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 20 08:49:20.873827 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 20 08:49:20.873827 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 20 08:49:20.873827 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 20 08:49:20.873827 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 20 08:49:20.873827 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application result = self._query(query) Oct 20 08:49:20.873827 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 20 08:49:20.873827 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application conn.query(q) Oct 20 08:49:20.873827 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 20 08:49:20.873827 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 20 08:49:20.873827 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 20 08:49:20.873827 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application result.read() Oct 20 08:49:20.874709 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 20 08:49:20.874709 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 20 08:49:20.874709 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 20 08:49:20.874709 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 20 08:49:20.874709 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 20 08:49:20.874709 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 20 08:49:20.874709 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 20 08:49:20.874709 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 20 08:49:20.874709 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 20 08:49:20.874709 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:49:20.874709 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 20 08:49:20.874709 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:49:20.874709 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:49:20.874709 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 20 08:49:20.874709 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 20 08:49:20.874709 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Oct 20 08:49:20.874709 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return project_ref.to_dict() Oct 20 08:49:20.874709 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 20 08:49:20.875696 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application next(self.gen) Oct 20 08:49:20.875696 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Oct 20 08:49:20.875696 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application yield resource Oct 20 08:49:20.875696 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 20 08:49:20.875696 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application next(self.gen) Oct 20 08:49:20.875696 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Oct 20 08:49:20.875696 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application self.session.rollback() Oct 20 08:49:20.875696 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 20 08:49:20.875696 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application self.force_reraise() Oct 20 08:49:20.875696 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 20 08:49:20.875696 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise self.value Oct 20 08:49:20.875696 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Oct 20 08:49:20.875696 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 20 08:49:20.875696 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Oct 20 08:49:20.875696 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application session.commit() Oct 20 08:49:20.875696 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 20 08:49:20.875696 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 20 08:49:20.876592 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 20 08:49:20.876592 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application self._prepare_impl() Oct 20 08:49:20.876592 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 20 08:49:20.876592 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application self.session.flush() Oct 20 08:49:20.876592 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 20 08:49:20.876592 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application self._flush(objects) Oct 20 08:49:20.876592 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 20 08:49:20.876592 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 20 08:49:20.876592 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 20 08:49:20.876592 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application compat.raise_( Oct 20 08:49:20.876592 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 20 08:49:20.876592 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception Oct 20 08:49:20.876592 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 20 08:49:20.876592 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application flush_context.execute() Oct 20 08:49:20.876592 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 20 08:49:20.876592 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rec.execute(self) Oct 20 08:49:20.876592 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 20 08:49:20.877510 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 20 08:49:20.877510 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 20 08:49:20.877510 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 20 08:49:20.877510 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Oct 20 08:49:20.877510 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application c = connection._execute_20( Oct 20 08:49:20.877510 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 20 08:49:20.877510 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 20 08:49:20.877510 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 20 08:49:20.877510 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 20 08:49:20.877510 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 20 08:49:20.877510 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 20 08:49:20.877510 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 20 08:49:20.877510 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 20 08:49:20.877510 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 20 08:49:20.877510 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 20 08:49:20.878323 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 20 08:49:20.878323 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception Oct 20 08:49:20.878323 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 20 08:49:20.878323 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 20 08:49:20.878323 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 20 08:49:20.878323 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 20 08:49:20.878323 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 20 08:49:20.878323 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application result = self._query(query) Oct 20 08:49:20.878323 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 20 08:49:20.878323 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application conn.query(q) Oct 20 08:49:20.878323 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 20 08:49:20.878323 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 20 08:49:20.878323 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 20 08:49:20.878323 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application result.read() Oct 20 08:49:20.878323 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 20 08:49:20.878323 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 20 08:49:20.878323 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 20 08:49:20.879273 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 20 08:49:20.879273 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 20 08:49:20.879273 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 20 08:49:20.879273 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 20 08:49:20.879273 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 20 08:49:20.879273 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 20 08:49:20.879273 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 20 08:49:20.879273 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [parameters: {'id': '3062eb2398aa4a5ebaebc95c47bb70a8', 'name': 'Default', 'domain_id': '<>', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': None, 'is_domain': 1}] Oct 20 08:49:20.879273 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 20 08:49:20.879273 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:49:20.879273 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 20 08:49:20.879273 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:49:20.879273 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:49:20.879273 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Oct 20 08:49:20.879273 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Oct 20 08:49:20.880121 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 20 08:49:20.880121 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 20 08:49:20.880121 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name Default. Oct 20 08:49:20.880121 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:49:20.880121 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 20 08:49:20.880121 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:49:20.880121 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:49:20.880121 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:49:20.880121 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:49:20.880121 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:49:20.880121 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:49:20.880121 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:49:20.880121 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:49:20.880121 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:49:20.880121 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:49:20.880121 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:49:20.880121 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:49:20.881046 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 128, in post Oct 20 08:49:20.881046 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_domain( Oct 20 08:49:20.881046 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:49:20.881046 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:49:20.881046 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 765, in create_domain Oct 20 08:49:20.881046 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application is_domain_project = self.create_project( Oct 20 08:49:20.881046 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:49:20.881046 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:49:20.881046 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Oct 20 08:49:20.881046 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.Conflict( Oct 20 08:49:20.881046 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default. Oct 20 08:49:20.881046 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:49:20.881046 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 406/800] 10.222.0.36 () {60 vars in 1137 bytes} [Thu Oct 20 08:49:20 2022] POST /identity/v3/domains => generated 190 bytes in 170 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:49:20.881046 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80390d94-8fb4-4bcb-9e9b-936575b13bf6 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:20.904797 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed05c36-ded7-42d0-9a87-0b5e1d854c7f None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:20.922539 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:20.940467 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:20.942951 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b90ca1f1-7b91-4033-a04d-15aaa7bcbb91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:20.943546 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90ca1f1-7b91-4033-a04d-15aaa7bcbb91 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:20.943818 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90ca1f1-7b91-4033-a04d-15aaa7bcbb91 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:20.944100 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90ca1f1-7b91-4033-a04d-15aaa7bcbb91 None admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:20.944789 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b90ca1f1-7b91-4033-a04d-15aaa7bcbb91 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:49:20.945085 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b90ca1f1-7b91-4033-a04d-15aaa7bcbb91 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:20.946547 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b90ca1f1-7b91-4033-a04d-15aaa7bcbb91 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:20.954960 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 395/801] 10.222.0.36 () {56 vars in 1115 bytes} [Thu Oct 20 08:49:20 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 08:49:20.960751 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b90ca1f1-7b91-4033-a04d-15aaa7bcbb91 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:20.992936 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80390d94-8fb4-4bcb-9e9b-936575b13bf6 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:21.026740 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbfb61aa-d1ae-47cc-8a64-8997243a3aa3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:21.027350 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbfb61aa-d1ae-47cc-8a64-8997243a3aa3 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:21.027623 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbfb61aa-d1ae-47cc-8a64-8997243a3aa3 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:21.027891 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbfb61aa-d1ae-47cc-8a64-8997243a3aa3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:21.039171 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbfb61aa-d1ae-47cc-8a64-8997243a3aa3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:21.040598 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbfb61aa-d1ae-47cc-8a64-8997243a3aa3 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:21.055296 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-dbfb61aa-d1ae-47cc-8a64-8997243a3aa3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:49:21.055296 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:49:21.055296 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:49:21.055296 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:49:21.055296 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:49:21.055296 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:49:21.055296 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:49:21.055296 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:49:21.055296 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:49:21.055296 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:49:21.055296 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:49:21.055296 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:49:21.055296 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 20 08:49:21.055296 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 20 08:49:21.055296 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 20 08:49:21.055296 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 20 08:49:21.056319 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:49:21.056319 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:49:21.056319 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:49:21.056319 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:49:21.056319 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:49:21.056319 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:49:21.056319 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:49:21.056319 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:49:21.056319 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:49:21.056319 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:49:21.056319 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:49:21.056319 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:49:21.056319 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:49:21.056319 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:49:21.056319 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:49:21.056319 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:49:21.056319 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 20 08:49:21.057233 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 20 08:49:21.057233 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:49:21.057233 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:49:21.057233 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 407/802] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:49:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 20 08:49:21.057233 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbfb61aa-d1ae-47cc-8a64-8997243a3aa3 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:21.063516 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b90ca1f1-7b91-4033-a04d-15aaa7bcbb91 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:21.099439 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-339ceb16-f38c-42b8-8754-f99ed3332219 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:21.100061 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-339ceb16-f38c-42b8-8754-f99ed3332219 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:21.100331 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-339ceb16-f38c-42b8-8754-f99ed3332219 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:21.100615 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-339ceb16-f38c-42b8-8754-f99ed3332219 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:21.111008 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-339ceb16-f38c-42b8-8754-f99ed3332219 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:21.112412 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-339ceb16-f38c-42b8-8754-f99ed3332219 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:21.126209 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-339ceb16-f38c-42b8-8754-f99ed3332219 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:49:21.126209 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:49:21.126209 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:49:21.126209 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:49:21.126209 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:49:21.126209 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:49:21.126209 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:49:21.126209 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:49:21.126209 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:49:21.126209 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:49:21.126209 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:49:21.126209 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:49:21.126209 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 20 08:49:21.126209 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 20 08:49:21.126209 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 20 08:49:21.126209 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 20 08:49:21.127279 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:49:21.127279 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:49:21.127279 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:49:21.127279 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:49:21.127279 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:49:21.127279 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:49:21.127279 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:49:21.127279 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:49:21.127279 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:49:21.127279 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:49:21.127279 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:49:21.127279 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:49:21.127279 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:49:21.127279 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:49:21.127279 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:49:21.127279 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:49:21.127279 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 20 08:49:21.128181 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 20 08:49:21.128181 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:49:21.128181 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:49:21.128181 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 396/803] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:49:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 20 08:49:21.128181 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-339ceb16-f38c-42b8-8754-f99ed3332219 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:21.161944 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbfb61aa-d1ae-47cc-8a64-8997243a3aa3 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:21.162123 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:21.170832 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:21.195091 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-100f95bf-2b0e-450c-85e2-93890054c593 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:21.195762 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100f95bf-2b0e-450c-85e2-93890054c593 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:21.196045 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100f95bf-2b0e-450c-85e2-93890054c593 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:21.196311 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100f95bf-2b0e-450c-85e2-93890054c593 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:21.206711 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-100f95bf-2b0e-450c-85e2-93890054c593 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 20 08:49:21.212674 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-100f95bf-2b0e-450c-85e2-93890054c593 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 20 08:49:21.212674 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:49:21.212674 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:49:21.212674 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:49:21.212674 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:49:21.212674 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:49:21.212674 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:49:21.212674 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:49:21.212674 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:49:21.212674 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:49:21.212674 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:49:21.212674 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:49:21.212674 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 20 08:49:21.212674 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 20 08:49:21.212674 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 20 08:49:21.212674 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 20 08:49:21.213801 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 20 08:49:21.213801 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 20 08:49:21.213801 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 20 08:49:21.213801 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 20 08:49:21.213801 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:49:21.213801 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:49:21.213801 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:49:21.213801 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:49:21.213801 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:49:21.213801 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:49:21.213801 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:49:21.213801 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:49:21.213801 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:49:21.213801 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:49:21.213801 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:49:21.213801 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:49:21.214960 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:49:21.214960 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:49:21.214960 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:49:21.214960 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:49:21.214960 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 20 08:49:21.214960 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 20 08:49:21.214960 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 20 08:49:21.214960 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 20 08:49:21.214960 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 20 08:49:21.214960 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 20 08:49:21.214960 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 20 08:49:21.214960 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:49:21.214960 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 408/804] 10.222.0.36 () {56 vars in 1096 bytes} [Thu Oct 20 08:49:21 2022] GET /identity/v3/roles/admin => generated 83 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:49:21.219366 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-100f95bf-2b0e-450c-85e2-93890054c593 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:21.232807 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-339ceb16-f38c-42b8-8754-f99ed3332219 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:21.264774 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb95d502-055c-41b8-94c7-4f59c795e92c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:21.265371 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb95d502-055c-41b8-94c7-4f59c795e92c None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:21.265902 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb95d502-055c-41b8-94c7-4f59c795e92c None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:21.265902 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb95d502-055c-41b8-94c7-4f59c795e92c None admin] PATH_INFO: `/v3/domains` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:21.266568 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb95d502-055c-41b8-94c7-4f59c795e92c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:49:21.266871 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb95d502-055c-41b8-94c7-4f59c795e92c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:21.268243 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb95d502-055c-41b8-94c7-4f59c795e92c None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:21.281588 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 397/805] 10.222.0.36 () {56 vars in 1109 bytes} [Thu Oct 20 08:49:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:49:21.282384 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb95d502-055c-41b8-94c7-4f59c795e92c None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:21.327628 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-100f95bf-2b0e-450c-85e2-93890054c593 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:21.368204 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b3fd4e-53eb-4229-9b2c-20a2dc2b30fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:21.368776 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b3fd4e-53eb-4229-9b2c-20a2dc2b30fe None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:21.369049 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b3fd4e-53eb-4229-9b2c-20a2dc2b30fe None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:21.369314 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b3fd4e-53eb-4229-9b2c-20a2dc2b30fe None admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:21.370003 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77b3fd4e-53eb-4229-9b2c-20a2dc2b30fe None admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:49:21.370309 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77b3fd4e-53eb-4229-9b2c-20a2dc2b30fe None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:21.371722 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77b3fd4e-53eb-4229-9b2c-20a2dc2b30fe None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:21.387980 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 409/806] 10.222.0.36 () {56 vars in 1109 bytes} [Thu Oct 20 08:49:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:49:21.393751 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b3fd4e-53eb-4229-9b2c-20a2dc2b30fe None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:21.406673 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:21.418038 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb95d502-055c-41b8-94c7-4f59c795e92c None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:21.426619 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:21.457186 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a95d7f6-911b-4778-99d6-e3046729b6d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:21.458445 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a95d7f6-911b-4778-99d6-e3046729b6d0 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:21.458755 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a95d7f6-911b-4778-99d6-e3046729b6d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:21.459028 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a95d7f6-911b-4778-99d6-e3046729b6d0 None admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:21.460570 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a95d7f6-911b-4778-99d6-e3046729b6d0 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:49:21.460570 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a95d7f6-911b-4778-99d6-e3046729b6d0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:21.462499 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a95d7f6-911b-4778-99d6-e3046729b6d0 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:21.470463 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 398/807] 10.222.0.36 () {56 vars in 1099 bytes} [Thu Oct 20 08:49:21 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:49:21.475982 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a95d7f6-911b-4778-99d6-e3046729b6d0 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:21.514820 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b3fd4e-53eb-4229-9b2c-20a2dc2b30fe None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:21.551961 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-651cf6a3-d355-4cc0-8993-913bcfb8dda8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:21.552592 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651cf6a3-d355-4cc0-8993-913bcfb8dda8 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:21.552870 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651cf6a3-d355-4cc0-8993-913bcfb8dda8 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:21.553868 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651cf6a3-d355-4cc0-8993-913bcfb8dda8 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:21.561294 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-651cf6a3-d355-4cc0-8993-913bcfb8dda8 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:21.562759 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-651cf6a3-d355-4cc0-8993-913bcfb8dda8 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:21.575181 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-651cf6a3-d355-4cc0-8993-913bcfb8dda8 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 20 08:49:21.575181 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:49:21.575181 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:49:21.575181 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:49:21.575181 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:49:21.575181 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:49:21.575181 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:49:21.575181 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:49:21.575181 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:49:21.575181 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:49:21.575181 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:49:21.575181 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:49:21.575181 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 20 08:49:21.575181 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 20 08:49:21.575181 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 20 08:49:21.575181 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 20 08:49:21.576189 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:49:21.576189 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:49:21.576189 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 20 08:49:21.576189 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 20 08:49:21.576189 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 20 08:49:21.576189 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 20 08:49:21.576189 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:49:21.576189 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:49:21.576189 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:49:21.576189 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:49:21.576189 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:49:21.576189 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:49:21.576189 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:49:21.576189 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:49:21.576189 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:49:21.576189 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:49:21.576189 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:49:21.576189 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:49:21.577159 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:49:21.577159 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:49:21.577159 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 20 08:49:21.577159 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 20 08:49:21.577159 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 20 08:49:21.577159 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 20 08:49:21.577159 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 20 08:49:21.577159 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 20 08:49:21.577159 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 20 08:49:21.577159 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:49:21.577159 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 410/808] 10.222.0.36 () {56 vars in 1093 bytes} [Thu Oct 20 08:49:21 2022] GET /identity/v3/users/nova => generated 82 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 20 08:49:21.582267 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-651cf6a3-d355-4cc0-8993-913bcfb8dda8 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:21.591217 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a95d7f6-911b-4778-99d6-e3046729b6d0 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:21.629064 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a6ef883-2c28-4762-a82f-6b1288379bd3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:21.630193 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a6ef883-2c28-4762-a82f-6b1288379bd3 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:21.630465 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a6ef883-2c28-4762-a82f-6b1288379bd3 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:21.630766 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a6ef883-2c28-4762-a82f-6b1288379bd3 None admin] PATH_INFO: `/v3/users/912e0c351efe402c9744857924bb4c64` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:21.643478 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a6ef883-2c28-4762-a82f-6b1288379bd3 None admin] RBAC: Authorizing `identity:get_user(user_id=912e0c351efe402c9744857924bb4c64)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:21.644935 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a6ef883-2c28-4762-a82f-6b1288379bd3 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:21.647246 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 399/809] 10.222.0.36 () {56 vars in 1177 bytes} [Thu Oct 20 08:49:21 2022] GET /identity/v3/users/912e0c351efe402c9744857924bb4c64 => generated 285 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 08:49:21.652982 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a6ef883-2c28-4762-a82f-6b1288379bd3 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:21.664353 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:21.676631 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:21.689343 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-651cf6a3-d355-4cc0-8993-913bcfb8dda8 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:21.723729 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df95017d-a088-47cb-b1a6-f7f13a290a2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:21.724327 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df95017d-a088-47cb-b1a6-f7f13a290a2a None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:21.724602 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df95017d-a088-47cb-b1a6-f7f13a290a2a None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:21.724868 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df95017d-a088-47cb-b1a6-f7f13a290a2a None admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:21.725581 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df95017d-a088-47cb-b1a6-f7f13a290a2a None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:49:21.725883 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df95017d-a088-47cb-b1a6-f7f13a290a2a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:21.727359 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df95017d-a088-47cb-b1a6-f7f13a290a2a None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:21.744009 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 411/810] 10.222.0.36 () {56 vars in 1133 bytes} [Thu Oct 20 08:49:21 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 08:49:21.749672 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df95017d-a088-47cb-b1a6-f7f13a290a2a None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:21.749955 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a6ef883-2c28-4762-a82f-6b1288379bd3 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:21.784083 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b38eafd5-b95d-4fc9-ab5d-536cf6144948 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:21.784697 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38eafd5-b95d-4fc9-ab5d-536cf6144948 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:21.784965 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38eafd5-b95d-4fc9-ab5d-536cf6144948 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:21.785255 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38eafd5-b95d-4fc9-ab5d-536cf6144948 None admin] PATH_INFO: `/v3/projects/041c141bd62d4c238c938ebf575f1653` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:21.786366 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b38eafd5-b95d-4fc9-ab5d-536cf6144948 None admin] RBAC: Authorizing `identity:get_project(project_id=041c141bd62d4c238c938ebf575f1653)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:21.787825 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b38eafd5-b95d-4fc9-ab5d-536cf6144948 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:21.789419 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 400/811] 10.222.0.36 () {56 vars in 1186 bytes} [Thu Oct 20 08:49:21 2022] GET /identity/v3/projects/041c141bd62d4c238c938ebf575f1653 => generated 311 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 08:49:21.795296 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b38eafd5-b95d-4fc9-ab5d-536cf6144948 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:21.848687 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df95017d-a088-47cb-b1a6-f7f13a290a2a None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:21.871361 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:21.883446 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abb60351-97c7-4dab-aadc-6ff2ae9e5fb7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:21.883997 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb60351-97c7-4dab-aadc-6ff2ae9e5fb7 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:21.884274 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb60351-97c7-4dab-aadc-6ff2ae9e5fb7 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:21.884543 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb60351-97c7-4dab-aadc-6ff2ae9e5fb7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:21.893017 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b38eafd5-b95d-4fc9-ab5d-536cf6144948 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:21.894689 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abb60351-97c7-4dab-aadc-6ff2ae9e5fb7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:21.896940 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abb60351-97c7-4dab-aadc-6ff2ae9e5fb7 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:21.904040 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:21.912506 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-abb60351-97c7-4dab-aadc-6ff2ae9e5fb7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:49:21.912506 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:49:21.912506 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:49:21.912506 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:49:21.912506 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:49:21.912506 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:49:21.912506 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:49:21.912506 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:49:21.912506 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:49:21.912506 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:49:21.912506 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:49:21.912506 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:49:21.912506 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 20 08:49:21.912506 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 20 08:49:21.912506 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 20 08:49:21.912506 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 20 08:49:21.913629 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:49:21.913629 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:49:21.913629 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:49:21.913629 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:49:21.913629 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:49:21.913629 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:49:21.913629 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:49:21.913629 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:49:21.913629 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:49:21.913629 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:49:21.913629 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:49:21.913629 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:49:21.913629 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:49:21.913629 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:49:21.913629 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:49:21.913629 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:49:21.913629 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 20 08:49:21.914765 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 20 08:49:21.914765 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:49:21.914765 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:49:21.914765 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 412/812] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:49:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 20 08:49:21.919119 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abb60351-97c7-4dab-aadc-6ff2ae9e5fb7 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:21.928835 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07c4fef8-36e6-43e2-829f-4122ce4da262 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:21.929478 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07c4fef8-36e6-43e2-829f-4122ce4da262 None admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:21.929754 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07c4fef8-36e6-43e2-829f-4122ce4da262 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:21.930020 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07c4fef8-36e6-43e2-829f-4122ce4da262 None admin] PATH_INFO: `/v3/projects/041c141bd62d4c238c938ebf575f1653/users/912e0c351efe402c9744857924bb4c64/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:21.931878 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07c4fef8-36e6-43e2-829f-4122ce4da262 None admin] RBAC: Authorizing `identity:create_grant(project_id=041c141bd62d4c238c938ebf575f1653, user_id=912e0c351efe402c9744857924bb4c64, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:21.933295 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07c4fef8-36e6-43e2-829f-4122ce4da262 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:21.946505 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 401/813] 10.222.0.36 () {58 vars in 1439 bytes} [Thu Oct 20 08:49:21 2022] PUT /identity/v3/projects/041c141bd62d4c238c938ebf575f1653/users/912e0c351efe402c9744857924bb4c64/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 08:49:22.022997 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abb60351-97c7-4dab-aadc-6ff2ae9e5fb7 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:22.058127 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8cc3bea-f9ae-4f69-89f2-fe4fcaddcada None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:22.058771 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8cc3bea-f9ae-4f69-89f2-fe4fcaddcada None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:22.059049 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8cc3bea-f9ae-4f69-89f2-fe4fcaddcada None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:22.059317 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8cc3bea-f9ae-4f69-89f2-fe4fcaddcada None admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:22.059965 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8cc3bea-f9ae-4f69-89f2-fe4fcaddcada None admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:49:22.060263 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8cc3bea-f9ae-4f69-89f2-fe4fcaddcada None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:22.061645 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8cc3bea-f9ae-4f69-89f2-fe4fcaddcada None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:22.076121 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 413/814] 10.222.0.36 () {56 vars in 1109 bytes} [Thu Oct 20 08:49:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 08:49:22.082412 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07c4fef8-36e6-43e2-829f-4122ce4da262 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:22.183017 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07c4fef8-36e6-43e2-829f-4122ce4da262 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:22.217136 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b0bd8c-27f7-4c09-9907-94c8f1ebe8bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:22.217755 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b0bd8c-27f7-4c09-9907-94c8f1ebe8bd None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:22.218029 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b0bd8c-27f7-4c09-9907-94c8f1ebe8bd None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:22.218298 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b0bd8c-27f7-4c09-9907-94c8f1ebe8bd None admin] PATH_INFO: `/v3/projects/service` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:22.225502 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06b0bd8c-27f7-4c09-9907-94c8f1ebe8bd None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:22.226966 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06b0bd8c-27f7-4c09-9907-94c8f1ebe8bd None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:22.228614 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:22.240791 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-06b0bd8c-27f7-4c09-9907-94c8f1ebe8bd None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 20 08:49:22.240791 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:49:22.240791 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:49:22.240791 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:49:22.240791 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:49:22.240791 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:49:22.240791 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:49:22.240791 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:49:22.240791 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:49:22.240791 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:49:22.240791 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:49:22.240791 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:49:22.240791 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 20 08:49:22.240791 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 20 08:49:22.240791 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 20 08:49:22.241699 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 20 08:49:22.241699 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:49:22.241699 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:49:22.241699 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:49:22.241699 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:49:22.241699 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:49:22.241699 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:49:22.241699 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:49:22.241699 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:49:22.241699 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:49:22.241699 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:49:22.241699 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:49:22.241699 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:49:22.241699 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:49:22.241699 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:49:22.241699 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:49:22.241699 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:49:22.242685 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 20 08:49:22.242685 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 20 08:49:22.242685 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 20 08:49:22.242685 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 20 08:49:22.242685 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 20 08:49:22.242685 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 20 08:49:22.242685 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 20 08:49:22.242685 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:49:22.242685 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 402/815] 10.222.0.36 () {56 vars in 1111 bytes} [Thu Oct 20 08:49:22 2022] GET /identity/v3/projects/service => generated 88 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:49:22.247780 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8cc3bea-f9ae-4f69-89f2-fe4fcaddcada None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:22.305254 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:22.345617 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8cc3bea-f9ae-4f69-89f2-fe4fcaddcada None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:22.383027 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6bccb1c-c7d4-4861-8cc5-5229d671c4f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:22.383630 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6bccb1c-c7d4-4861-8cc5-5229d671c4f0 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:22.383905 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6bccb1c-c7d4-4861-8cc5-5229d671c4f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:22.384172 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6bccb1c-c7d4-4861-8cc5-5229d671c4f0 None admin] PATH_INFO: `/v3/projects` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:22.384859 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6bccb1c-c7d4-4861-8cc5-5229d671c4f0 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:49:22.385160 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6bccb1c-c7d4-4861-8cc5-5229d671c4f0 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:22.386596 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6bccb1c-c7d4-4861-8cc5-5229d671c4f0 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:22.398837 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 414/816] 10.222.0.36 () {56 vars in 1148 bytes} [Thu Oct 20 08:49:22 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:49:22.404749 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b0bd8c-27f7-4c09-9907-94c8f1ebe8bd None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:22.502748 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b0bd8c-27f7-4c09-9907-94c8f1ebe8bd None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:22.535962 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61ccc67d-555c-4fdb-baae-5e6c8f7bcd0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:22.536603 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ccc67d-555c-4fdb-baae-5e6c8f7bcd0f None admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:22.536875 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ccc67d-555c-4fdb-baae-5e6c8f7bcd0f None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:22.537141 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ccc67d-555c-4fdb-baae-5e6c8f7bcd0f None admin] PATH_INFO: `/v3/projects/d9857b610fb542969cfb5efbbdb8ce61/users/a94e868115ec49918eda10724263930c/roles/37a51151d3c042ffbaca4dc2bf317baf` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:22.539312 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61ccc67d-555c-4fdb-baae-5e6c8f7bcd0f None admin] RBAC: Authorizing `identity:create_grant(project_id=d9857b610fb542969cfb5efbbdb8ce61, user_id=a94e868115ec49918eda10724263930c, role_id=37a51151d3c042ffbaca4dc2bf317baf)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:22.540725 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61ccc67d-555c-4fdb-baae-5e6c8f7bcd0f None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:22.553807 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 403/817] 10.222.0.36 () {58 vars in 1439 bytes} [Thu Oct 20 08:49:22 2022] PUT /identity/v3/projects/d9857b610fb542969cfb5efbbdb8ce61/users/a94e868115ec49918eda10724263930c/roles/37a51151d3c042ffbaca4dc2bf317baf => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 08:49:24.072248 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc8f7039-a3df-46b4-88fc-67af5e058fb1 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:24.072869 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc8f7039-a3df-46b4-88fc-67af5e058fb1 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:24.072972 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc8f7039-a3df-46b4-88fc-67af5e058fb1 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:24.074491 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 415/818] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:49:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:49:24.079954 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8bf2967-ad5a-4c62-bd18-7c00d670abaa None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:24.080365 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8bf2967-ad5a-4c62-bd18-7c00d670abaa None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:24.080474 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8bf2967-ad5a-4c62-bd18-7c00d670abaa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:24.098179 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-a8bf2967-ad5a-4c62-bd18-7c00d670abaa None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:49:24.099367 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-a8bf2967-ad5a-4c62-bd18-7c00d670abaa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:49:24.151043 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:24.166595 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 404/819] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:49:24 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 20 08:49:24.432040 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc8f7039-a3df-46b4-88fc-67af5e058fb1 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:24.532241 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc8f7039-a3df-46b4-88fc-67af5e058fb1 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:24.550877 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:24.569250 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6276b04a-3af7-4d6a-b2f0-1aa30ed89ff3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:24.569894 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6276b04a-3af7-4d6a-b2f0-1aa30ed89ff3 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:24.570170 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6276b04a-3af7-4d6a-b2f0-1aa30ed89ff3 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:24.570462 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6276b04a-3af7-4d6a-b2f0-1aa30ed89ff3 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:24.581999 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6276b04a-3af7-4d6a-b2f0-1aa30ed89ff3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 20 08:49:24.587240 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-6276b04a-3af7-4d6a-b2f0-1aa30ed89ff3 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 20 08:49:24.587240 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:49:24.587240 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:49:24.587240 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:49:24.587240 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:49:24.587240 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:49:24.587240 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:49:24.587240 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:49:24.587240 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:49:24.587240 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:49:24.587240 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:49:24.587240 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:49:24.587240 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 20 08:49:24.587240 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 20 08:49:24.587240 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 20 08:49:24.587240 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 20 08:49:24.588331 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 20 08:49:24.588331 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 20 08:49:24.588331 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 20 08:49:24.588331 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 20 08:49:24.588331 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:49:24.588331 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:49:24.588331 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:49:24.588331 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:49:24.588331 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:49:24.588331 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:49:24.588331 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:49:24.588331 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:49:24.588331 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:49:24.588331 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:49:24.588331 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:49:24.588331 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:49:24.589245 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:49:24.589245 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:49:24.589245 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:49:24.589245 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:49:24.589245 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 20 08:49:24.589245 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 20 08:49:24.589245 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 20 08:49:24.589245 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 20 08:49:24.589245 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 20 08:49:24.589245 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 20 08:49:24.589245 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 20 08:49:24.589245 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:49:24.589245 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 416/820] 10.222.0.36 () {56 vars in 1096 bytes} [Thu Oct 20 08:49:24 2022] GET /identity/v3/roles/admin => generated 83 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:49:24.594458 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8bf2967-ad5a-4c62-bd18-7c00d670abaa None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:24.664711 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8605fd7a-12d8-4e21-8cc9-9e536de8c5d4 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:24.664906 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8605fd7a-12d8-4e21-8cc9-9e536de8c5d4 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:24.665165 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8605fd7a-12d8-4e21-8cc9-9e536de8c5d4 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:24.666494 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 417/821] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:49:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 20 08:49:24.671356 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7fd9f0a-6cfc-489d-ac0f-b1e7cc3b73f4 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:24.671633 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7fd9f0a-6cfc-489d-ac0f-b1e7cc3b73f4 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:24.671898 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7fd9f0a-6cfc-489d-ac0f-b1e7cc3b73f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:24.690082 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-c7fd9f0a-6cfc-489d-ac0f-b1e7cc3b73f4 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:49:24.691298 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-c7fd9f0a-6cfc-489d-ac0f-b1e7cc3b73f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:49:24.691499 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8bf2967-ad5a-4c62-bd18-7c00d670abaa None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:24.724679 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b18607-0a4d-4cd0-b039-01770b0972d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:24.725267 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b18607-0a4d-4cd0-b039-01770b0972d2 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:24.725538 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b18607-0a4d-4cd0-b039-01770b0972d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:24.725800 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b18607-0a4d-4cd0-b039-01770b0972d2 None admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:24.726448 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7b18607-0a4d-4cd0-b039-01770b0972d2 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:49:24.726773 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7b18607-0a4d-4cd0-b039-01770b0972d2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:24.728124 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7b18607-0a4d-4cd0-b039-01770b0972d2 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:24.736079 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 405/822] 10.222.0.36 () {56 vars in 1099 bytes} [Thu Oct 20 08:49:24 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:49:24.742591 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b18607-0a4d-4cd0-b039-01770b0972d2 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:24.757225 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 418/823] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:49:24 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 20 08:49:24.789948 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:24.839362 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b18607-0a4d-4cd0-b039-01770b0972d2 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:24.871951 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d873cc-b01a-4ce1-8f2e-fb738bebd87d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:24.872566 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d873cc-b01a-4ce1-8f2e-fb738bebd87d None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:24.872843 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d873cc-b01a-4ce1-8f2e-fb738bebd87d None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:24.873110 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d873cc-b01a-4ce1-8f2e-fb738bebd87d None admin] PATH_INFO: `/v3/users/912e0c351efe402c9744857924bb4c64` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:24.874214 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54d873cc-b01a-4ce1-8f2e-fb738bebd87d None admin] RBAC: Authorizing `identity:get_user(user_id=912e0c351efe402c9744857924bb4c64)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:24.875688 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54d873cc-b01a-4ce1-8f2e-fb738bebd87d None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:24.877250 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 406/824] 10.222.0.36 () {56 vars in 1177 bytes} [Thu Oct 20 08:49:24 2022] GET /identity/v3/users/912e0c351efe402c9744857924bb4c64 => generated 285 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 08:49:24.883029 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7fd9f0a-6cfc-489d-ac0f-b1e7cc3b73f4 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:24.977367 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7fd9f0a-6cfc-489d-ac0f-b1e7cc3b73f4 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:24.986105 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:25.009795 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12137374-26a7-45de-a1da-417eea3b6d04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:25.010386 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12137374-26a7-45de-a1da-417eea3b6d04 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:25.010684 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12137374-26a7-45de-a1da-417eea3b6d04 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:25.010948 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12137374-26a7-45de-a1da-417eea3b6d04 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:25.025150 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d873cc-b01a-4ce1-8f2e-fb738bebd87d None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:25.075567 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12137374-26a7-45de-a1da-417eea3b6d04 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:25.076999 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12137374-26a7-45de-a1da-417eea3b6d04 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:25.123538 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d873cc-b01a-4ce1-8f2e-fb738bebd87d None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:25.139575 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:25.158148 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cdfe0d5-b720-43cc-a832-32b3ad8ff394 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:25.158795 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cdfe0d5-b720-43cc-a832-32b3ad8ff394 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:25.159070 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cdfe0d5-b720-43cc-a832-32b3ad8ff394 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:25.159337 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cdfe0d5-b720-43cc-a832-32b3ad8ff394 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:25.170441 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cdfe0d5-b720-43cc-a832-32b3ad8ff394 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 20 08:49:25.173448 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 419/825] 10.222.0.36 () {58 vars in 1282 bytes} [Thu Oct 20 08:49:24 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 291 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 20 08:49:25.175152 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-4cdfe0d5-b720-43cc-a832-32b3ad8ff394 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 20 08:49:25.175152 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:49:25.175152 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:49:25.175152 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:49:25.175152 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:49:25.175152 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:49:25.175152 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:49:25.175152 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:49:25.175152 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:49:25.175152 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:49:25.175152 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:49:25.175152 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:49:25.175152 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 20 08:49:25.175152 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 20 08:49:25.175152 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 20 08:49:25.175152 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 20 08:49:25.176246 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 20 08:49:25.176246 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 20 08:49:25.176246 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 20 08:49:25.176246 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 20 08:49:25.176246 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:49:25.176246 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:49:25.176246 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:49:25.176246 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:49:25.176246 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:49:25.176246 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:49:25.176246 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:49:25.176246 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:49:25.176246 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:49:25.176246 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:49:25.176246 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:49:25.176246 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:49:25.177302 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:49:25.177302 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:49:25.177302 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:49:25.177302 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:49:25.177302 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 20 08:49:25.177302 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 20 08:49:25.177302 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 20 08:49:25.177302 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 20 08:49:25.177302 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 20 08:49:25.177302 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 20 08:49:25.177302 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 20 08:49:25.177302 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:49:25.177302 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 407/826] 10.222.0.36 () {56 vars in 1120 bytes} [Thu Oct 20 08:49:25 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 20 08:49:25.179461 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12137374-26a7-45de-a1da-417eea3b6d04 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:25.181752 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cdfe0d5-b720-43cc-a832-32b3ad8ff394 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:25.188916 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:25.277791 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12137374-26a7-45de-a1da-417eea3b6d04 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:25.279081 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cdfe0d5-b720-43cc-a832-32b3ad8ff394 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:25.310975 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9f66b04-2737-4f78-a7ee-a85b9ec85220 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:25.311252 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e480b87c-4711-4b22-99b8-52fd29b471b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:25.311563 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f66b04-2737-4f78-a7ee-a85b9ec85220 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:25.311730 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e480b87c-4711-4b22-99b8-52fd29b471b3 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:25.311828 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f66b04-2737-4f78-a7ee-a85b9ec85220 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:25.312003 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e480b87c-4711-4b22-99b8-52fd29b471b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:25.312101 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f66b04-2737-4f78-a7ee-a85b9ec85220 None admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:25.312269 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e480b87c-4711-4b22-99b8-52fd29b471b3 None admin] PATH_INFO: `/v3/projects/041c141bd62d4c238c938ebf575f1653` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:25.312746 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9f66b04-2737-4f78-a7ee-a85b9ec85220 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:49:25.313026 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9f66b04-2737-4f78-a7ee-a85b9ec85220 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:25.313375 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e480b87c-4711-4b22-99b8-52fd29b471b3 None admin] RBAC: Authorizing `identity:get_project(project_id=041c141bd62d4c238c938ebf575f1653)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:25.314382 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9f66b04-2737-4f78-a7ee-a85b9ec85220 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:25.314801 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e480b87c-4711-4b22-99b8-52fd29b471b3 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:25.316608 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 420/827] 10.222.0.36 () {56 vars in 1186 bytes} [Thu Oct 20 08:49:25 2022] GET /identity/v3/projects/041c141bd62d4c238c938ebf575f1653 => generated 311 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 08:49:25.322851 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e480b87c-4711-4b22-99b8-52fd29b471b3 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:25.322992 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 408/828] 10.222.0.36 () {56 vars in 1115 bytes} [Thu Oct 20 08:49:25 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 08:49:25.328627 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9f66b04-2737-4f78-a7ee-a85b9ec85220 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:25.369384 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:25.402505 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:25.425760 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e480b87c-4711-4b22-99b8-52fd29b471b3 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:25.428989 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9f66b04-2737-4f78-a7ee-a85b9ec85220 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:25.459201 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc7f6ca7-7f33-479a-bd52-d5473892b5d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:25.460044 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7f6ca7-7f33-479a-bd52-d5473892b5d1 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:25.460318 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7f6ca7-7f33-479a-bd52-d5473892b5d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:25.460585 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7f6ca7-7f33-479a-bd52-d5473892b5d1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:25.461300 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc7f6ca7-7f33-479a-bd52-d5473892b5d1 None admin] RBAC: Adding query filter params (role.id=d9d3bc62f6e8453488c74856ef333bb3, scope.project.id=041c141bd62d4c238c938ebf575f1653, user.id=912e0c351efe402c9744857924bb4c64) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:49:25.461603 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc7f6ca7-7f33-479a-bd52-d5473892b5d1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:25.462127 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b938f5db-93f3-41a8-8b01-3e1df51006b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:25.462769 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b938f5db-93f3-41a8-8b01-3e1df51006b2 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:25.463044 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b938f5db-93f3-41a8-8b01-3e1df51006b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:25.463142 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc7f6ca7-7f33-479a-bd52-d5473892b5d1 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:25.463305 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b938f5db-93f3-41a8-8b01-3e1df51006b2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:25.468719 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 421/829] 10.222.0.36 () {56 vars in 1374 bytes} [Thu Oct 20 08:49:25 2022] GET /identity/v3/role_assignments?user.id=912e0c351efe402c9744857924bb4c64&scope.project.id=041c141bd62d4c238c938ebf575f1653&role.id=d9d3bc62f6e8453488c74856ef333bb3 => generated 608 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 08:49:25.473611 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b938f5db-93f3-41a8-8b01-3e1df51006b2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:25.475393 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b938f5db-93f3-41a8-8b01-3e1df51006b2 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:25.491496 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-b938f5db-93f3-41a8-8b01-3e1df51006b2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:49:25.491496 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:49:25.491496 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:49:25.491496 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:49:25.491496 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:49:25.491496 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:49:25.491496 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:49:25.491496 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:49:25.491496 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:49:25.491496 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:49:25.491496 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:49:25.491496 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:49:25.491496 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 20 08:49:25.491496 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 20 08:49:25.491496 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 20 08:49:25.491496 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 20 08:49:25.492747 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:49:25.492747 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:49:25.492747 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:49:25.492747 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:49:25.492747 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:49:25.492747 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:49:25.492747 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:49:25.492747 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:49:25.492747 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:49:25.492747 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:49:25.492747 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:49:25.492747 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:49:25.492747 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:49:25.492747 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:49:25.492747 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:49:25.492747 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:49:25.492747 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 20 08:49:25.493678 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 20 08:49:25.493678 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:49:25.493678 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:49:25.493678 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 409/830] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:49:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 20 08:49:25.498938 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc7f6ca7-7f33-479a-bd52-d5473892b5d1 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:25.718775 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc7f6ca7-7f33-479a-bd52-d5473892b5d1 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:25.751720 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-705ab8e1-cbce-4cc2-a5aa-d43f1a0e520a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:25.752392 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705ab8e1-cbce-4cc2-a5aa-d43f1a0e520a None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:25.752672 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705ab8e1-cbce-4cc2-a5aa-d43f1a0e520a None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:25.752944 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705ab8e1-cbce-4cc2-a5aa-d43f1a0e520a None admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:25.753688 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-705ab8e1-cbce-4cc2-a5aa-d43f1a0e520a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:49:25.753988 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-705ab8e1-cbce-4cc2-a5aa-d43f1a0e520a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:25.755448 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-705ab8e1-cbce-4cc2-a5aa-d43f1a0e520a None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:25.764108 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:25.769018 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 422/831] 10.222.0.36 () {56 vars in 1109 bytes} [Thu Oct 20 08:49:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 271 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:49:25.775047 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b938f5db-93f3-41a8-8b01-3e1df51006b2 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:25.865329 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:25.874453 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b938f5db-93f3-41a8-8b01-3e1df51006b2 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:25.907341 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ad79276-95b8-4c51-a2ab-051963f7a01d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:25.907957 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ad79276-95b8-4c51-a2ab-051963f7a01d None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:25.908232 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ad79276-95b8-4c51-a2ab-051963f7a01d None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:25.908501 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ad79276-95b8-4c51-a2ab-051963f7a01d None admin] PATH_INFO: `/v3/users/nova` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:25.915860 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ad79276-95b8-4c51-a2ab-051963f7a01d None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:25.917297 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ad79276-95b8-4c51-a2ab-051963f7a01d None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:25.928755 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-5ad79276-95b8-4c51-a2ab-051963f7a01d None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 20 08:49:25.928755 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:49:25.928755 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:49:25.928755 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:49:25.928755 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:49:25.928755 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:49:25.928755 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:49:25.928755 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:49:25.928755 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:49:25.928755 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:49:25.928755 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:49:25.928755 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:49:25.928755 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 20 08:49:25.928755 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 20 08:49:25.928755 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 20 08:49:25.928755 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 20 08:49:25.929798 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:49:25.929798 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:49:25.929798 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 20 08:49:25.929798 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 20 08:49:25.929798 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 20 08:49:25.929798 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 20 08:49:25.929798 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:49:25.929798 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:49:25.929798 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:49:25.929798 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:49:25.929798 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:49:25.929798 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:49:25.929798 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:49:25.929798 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:49:25.929798 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:49:25.929798 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:49:25.929798 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:49:25.929798 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:49:25.931148 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:49:25.931148 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:49:25.931148 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 20 08:49:25.931148 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 20 08:49:25.931148 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 20 08:49:25.931148 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 20 08:49:25.931148 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 20 08:49:25.931148 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 20 08:49:25.931148 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 20 08:49:25.931148 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:49:25.931148 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 410/832] 10.222.0.36 () {56 vars in 1093 bytes} [Thu Oct 20 08:49:25 2022] GET /identity/v3/users/nova => generated 82 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:49:25.935634 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-705ab8e1-cbce-4cc2-a5aa-d43f1a0e520a None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:26.033312 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-705ab8e1-cbce-4cc2-a5aa-d43f1a0e520a None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:26.065891 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-199c91c8-b8fc-40e6-90ab-7213157fdf0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:26.066501 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-199c91c8-b8fc-40e6-90ab-7213157fdf0c None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:26.066787 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-199c91c8-b8fc-40e6-90ab-7213157fdf0c None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:26.067088 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-199c91c8-b8fc-40e6-90ab-7213157fdf0c None admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:26.067798 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-199c91c8-b8fc-40e6-90ab-7213157fdf0c None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:49:26.068101 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-199c91c8-b8fc-40e6-90ab-7213157fdf0c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:26.069518 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-199c91c8-b8fc-40e6-90ab-7213157fdf0c None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:26.086432 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 423/833] 10.222.0.36 () {56 vars in 1133 bytes} [Thu Oct 20 08:49:25 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:49:26.091925 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ad79276-95b8-4c51-a2ab-051963f7a01d None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:26.191114 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ad79276-95b8-4c51-a2ab-051963f7a01d None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:26.225083 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f34456c7-efe8-4e43-8730-7d9082e41b29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:26.225692 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f34456c7-efe8-4e43-8730-7d9082e41b29 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:26.225970 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f34456c7-efe8-4e43-8730-7d9082e41b29 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:26.226242 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f34456c7-efe8-4e43-8730-7d9082e41b29 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:26.247432 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:26.292635 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f34456c7-efe8-4e43-8730-7d9082e41b29 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:26.294016 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f34456c7-efe8-4e43-8730-7d9082e41b29 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:26.395994 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 411/834] 10.222.0.36 () {58 vars in 1282 bytes} [Thu Oct 20 08:49:26 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 20 08:49:26.401828 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-199c91c8-b8fc-40e6-90ab-7213157fdf0c None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:26.458423 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:26.498832 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-199c91c8-b8fc-40e6-90ab-7213157fdf0c None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:26.531242 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-890d65aa-6744-4c03-bdb8-b0f68ccbb2cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:26.531846 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-890d65aa-6744-4c03-bdb8-b0f68ccbb2cb None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:26.532123 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-890d65aa-6744-4c03-bdb8-b0f68ccbb2cb None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:26.532391 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-890d65aa-6744-4c03-bdb8-b0f68ccbb2cb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:26.542020 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-890d65aa-6744-4c03-bdb8-b0f68ccbb2cb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:26.543528 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-890d65aa-6744-4c03-bdb8-b0f68ccbb2cb None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:26.557902 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-890d65aa-6744-4c03-bdb8-b0f68ccbb2cb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:49:26.557902 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:49:26.557902 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:49:26.557902 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:49:26.557902 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:49:26.557902 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:49:26.557902 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:49:26.557902 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:49:26.557902 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:49:26.557902 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:49:26.557902 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:49:26.557902 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:49:26.557902 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 20 08:49:26.557902 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 20 08:49:26.557902 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 20 08:49:26.557902 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 20 08:49:26.563604 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:49:26.563604 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:49:26.563604 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:49:26.563604 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:49:26.563604 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:49:26.563604 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:49:26.563604 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:49:26.563604 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:49:26.563604 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:49:26.563604 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:49:26.563604 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:49:26.563604 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:49:26.563604 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:49:26.563604 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:49:26.563604 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:49:26.563604 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:49:26.563604 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 20 08:49:26.564556 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 20 08:49:26.564556 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:49:26.564556 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:49:26.564556 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 424/835] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:49:26 2022] GET /identity/v3/domains/Default => generated 87 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:49:26.564780 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f34456c7-efe8-4e43-8730-7d9082e41b29 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:26.617041 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:26.663831 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f34456c7-efe8-4e43-8730-7d9082e41b29 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:26.697466 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de3fa9a-049c-429d-91f5-6024c81e3d05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:26.698062 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de3fa9a-049c-429d-91f5-6024c81e3d05 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:26.698330 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de3fa9a-049c-429d-91f5-6024c81e3d05 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:26.698620 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de3fa9a-049c-429d-91f5-6024c81e3d05 None admin] PATH_INFO: `/v3/domains` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:26.699274 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4de3fa9a-049c-429d-91f5-6024c81e3d05 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:49:26.699566 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4de3fa9a-049c-429d-91f5-6024c81e3d05 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:26.700914 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4de3fa9a-049c-429d-91f5-6024c81e3d05 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:26.712139 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 412/836] 10.222.0.36 () {56 vars in 1109 bytes} [Thu Oct 20 08:49:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 08:49:26.717796 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-890d65aa-6744-4c03-bdb8-b0f68ccbb2cb None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:26.814137 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-890d65aa-6744-4c03-bdb8-b0f68ccbb2cb None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:26.824162 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:26.846906 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97a9fee0-fba8-4809-9902-6ba9761c38a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:26.847521 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a9fee0-fba8-4809-9902-6ba9761c38a8 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:26.847795 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a9fee0-fba8-4809-9902-6ba9761c38a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:26.848063 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a9fee0-fba8-4809-9902-6ba9761c38a8 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:26.855069 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97a9fee0-fba8-4809-9902-6ba9761c38a8 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:26.856489 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97a9fee0-fba8-4809-9902-6ba9761c38a8 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:26.868667 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-97a9fee0-fba8-4809-9902-6ba9761c38a8 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 20 08:49:26.868667 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:49:26.868667 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:49:26.868667 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:49:26.868667 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:49:26.868667 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:49:26.868667 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:49:26.868667 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:49:26.868667 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:49:26.868667 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:49:26.868667 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:49:26.868667 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:49:26.868667 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 20 08:49:26.868667 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 20 08:49:26.868667 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 20 08:49:26.869575 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 20 08:49:26.869575 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:49:26.869575 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:49:26.869575 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:49:26.869575 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:49:26.869575 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:49:26.869575 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:49:26.869575 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:49:26.869575 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:49:26.869575 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:49:26.869575 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:49:26.869575 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:49:26.869575 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:49:26.869575 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:49:26.869575 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:49:26.869575 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:49:26.869575 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:49:26.870851 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 20 08:49:26.870851 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 20 08:49:26.870851 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 20 08:49:26.870851 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 20 08:49:26.870851 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 20 08:49:26.870851 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 20 08:49:26.870851 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 20 08:49:26.870851 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:49:26.870851 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 425/837] 10.222.0.36 () {56 vars in 1111 bytes} [Thu Oct 20 08:49:26 2022] GET /identity/v3/projects/service => generated 88 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:49:26.875571 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de3fa9a-049c-429d-91f5-6024c81e3d05 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:26.970896 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de3fa9a-049c-429d-91f5-6024c81e3d05 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:26.982849 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:27.002926 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76da6467-4f3c-4ea2-9acd-511c3e0eda38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:27.003516 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76da6467-4f3c-4ea2-9acd-511c3e0eda38 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:27.003788 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76da6467-4f3c-4ea2-9acd-511c3e0eda38 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:27.004051 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76da6467-4f3c-4ea2-9acd-511c3e0eda38 None admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:27.005275 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76da6467-4f3c-4ea2-9acd-511c3e0eda38 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:49:27.005571 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76da6467-4f3c-4ea2-9acd-511c3e0eda38 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:27.007018 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76da6467-4f3c-4ea2-9acd-511c3e0eda38 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:27.018501 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 413/838] 10.222.0.36 () {56 vars in 1148 bytes} [Thu Oct 20 08:49:26 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:49:27.025097 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97a9fee0-fba8-4809-9902-6ba9761c38a8 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:27.121498 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97a9fee0-fba8-4809-9902-6ba9761c38a8 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:27.154478 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2972da0f-024f-4ff3-8f4a-86f5a20039f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:27.155096 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2972da0f-024f-4ff3-8f4a-86f5a20039f1 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:27.155373 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2972da0f-024f-4ff3-8f4a-86f5a20039f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:27.155642 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2972da0f-024f-4ff3-8f4a-86f5a20039f1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:27.156353 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2972da0f-024f-4ff3-8f4a-86f5a20039f1 None admin] RBAC: Adding query filter params (role.id=37a51151d3c042ffbaca4dc2bf317baf, scope.project.id=d9857b610fb542969cfb5efbbdb8ce61, user.id=a94e868115ec49918eda10724263930c) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:49:27.156656 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2972da0f-024f-4ff3-8f4a-86f5a20039f1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:27.158074 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2972da0f-024f-4ff3-8f4a-86f5a20039f1 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:27.164322 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 426/839] 10.222.0.36 () {56 vars in 1374 bytes} [Thu Oct 20 08:49:27 2022] GET /identity/v3/role_assignments?user.id=a94e868115ec49918eda10724263930c&scope.project.id=d9857b610fb542969cfb5efbbdb8ce61&role.id=37a51151d3c042ffbaca4dc2bf317baf => generated 608 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:49:27.652110 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17be101-e074-41c3-9125-f16ae707be46 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:27.652601 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17be101-e074-41c3-9125-f16ae707be46 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:27.652601 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17be101-e074-41c3-9125-f16ae707be46 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:27.653901 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 414/840] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:49:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:49:27.659391 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b378474-691a-4e08-8fcd-f22ac7bb55cc None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:27.659679 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b378474-691a-4e08-8fcd-f22ac7bb55cc None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:27.659949 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b378474-691a-4e08-8fcd-f22ac7bb55cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:27.679388 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-2b378474-691a-4e08-8fcd-f22ac7bb55cc None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:49:27.680558 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-2b378474-691a-4e08-8fcd-f22ac7bb55cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:49:27.706755 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:27.748824 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 427/841] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:49:27 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 20 08:49:28.018153 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e17be101-e074-41c3-9125-f16ae707be46 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:28.117124 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e17be101-e074-41c3-9125-f16ae707be46 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:28.149938 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-948aadf0-fa15-4896-ae98-7d522a1c6e53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:28.150575 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948aadf0-fa15-4896-ae98-7d522a1c6e53 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:28.150856 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948aadf0-fa15-4896-ae98-7d522a1c6e53 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:28.151125 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948aadf0-fa15-4896-ae98-7d522a1c6e53 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:28.152215 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-948aadf0-fa15-4896-ae98-7d522a1c6e53 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:28.153645 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-948aadf0-fa15-4896-ae98-7d522a1c6e53 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:28.155204 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 415/842] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:49:28 2022] GET /identity/v3/domains/default => generated 199 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:49:28.160885 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b378474-691a-4e08-8fcd-f22ac7bb55cc None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:28.256127 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b378474-691a-4e08-8fcd-f22ac7bb55cc None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:28.288667 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79588ab9-6fa6-4a35-9d79-8c52a1abf2ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:28.289194 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79588ab9-6fa6-4a35-9d79-8c52a1abf2ad None admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:28.289466 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79588ab9-6fa6-4a35-9d79-8c52a1abf2ad None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:28.289730 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79588ab9-6fa6-4a35-9d79-8c52a1abf2ad None admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:28.290372 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79588ab9-6fa6-4a35-9d79-8c52a1abf2ad None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:28.291798 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79588ab9-6fa6-4a35-9d79-8c52a1abf2ad None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:28.312990 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 428/843] 10.222.0.36 () {60 vars in 1132 bytes} [Thu Oct 20 08:49:28 2022] POST /identity/v3/users => generated 286 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 08:49:30.431801 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b006124b-66aa-45d5-b93a-d95d45fc3e12 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:30.432243 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b006124b-66aa-45d5-b93a-d95d45fc3e12 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:30.432351 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b006124b-66aa-45d5-b93a-d95d45fc3e12 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:30.433664 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 416/844] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:49:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:49:30.438839 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e306dba2-ef41-4629-a7b4-bdabfab6c781 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:30.439258 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e306dba2-ef41-4629-a7b4-bdabfab6c781 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:30.439383 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e306dba2-ef41-4629-a7b4-bdabfab6c781 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:30.458228 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-e306dba2-ef41-4629-a7b4-bdabfab6c781 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:49:30.459436 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-e306dba2-ef41-4629-a7b4-bdabfab6c781 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:49:30.472930 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:30.523183 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 429/845] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:49:30 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 20 08:49:30.788726 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b006124b-66aa-45d5-b93a-d95d45fc3e12 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:30.830183 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:30.886956 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b006124b-66aa-45d5-b93a-d95d45fc3e12 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:30.918533 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c503a77e-ca66-4d0b-a8ac-d07f9d37a09c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:30.919141 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c503a77e-ca66-4d0b-a8ac-d07f9d37a09c None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:30.919417 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c503a77e-ca66-4d0b-a8ac-d07f9d37a09c None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:30.919684 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c503a77e-ca66-4d0b-a8ac-d07f9d37a09c None admin] PATH_INFO: `/v3/roles/b3f8d63bf4c4488288e57693c6e9804c` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:30.921050 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c503a77e-ca66-4d0b-a8ac-d07f9d37a09c None admin] RBAC: Authorizing `identity:get_role(role_id=b3f8d63bf4c4488288e57693c6e9804c)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:30.922402 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c503a77e-ca66-4d0b-a8ac-d07f9d37a09c None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:30.923721 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 417/846] 10.222.0.36 () {56 vars in 1177 bytes} [Thu Oct 20 08:49:30 2022] GET /identity/v3/roles/b3f8d63bf4c4488288e57693c6e9804c => generated 223 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:49:30.929837 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e306dba2-ef41-4629-a7b4-bdabfab6c781 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:31.023295 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e306dba2-ef41-4629-a7b4-bdabfab6c781 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:31.055266 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a74e1e21-fc0b-46e3-bd27-9aa74125a80b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:31.055815 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a74e1e21-fc0b-46e3-bd27-9aa74125a80b None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:31.056086 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a74e1e21-fc0b-46e3-bd27-9aa74125a80b None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:31.056348 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a74e1e21-fc0b-46e3-bd27-9aa74125a80b None admin] PATH_INFO: `/v3/users/edf8c5f1c0294286818fe32c699da63b` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:31.069477 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a74e1e21-fc0b-46e3-bd27-9aa74125a80b None admin] RBAC: Authorizing `identity:get_user(user_id=edf8c5f1c0294286818fe32c699da63b)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:31.070923 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a74e1e21-fc0b-46e3-bd27-9aa74125a80b None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:31.072470 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 430/847] 10.222.0.36 () {56 vars in 1177 bytes} [Thu Oct 20 08:49:30 2022] GET /identity/v3/users/edf8c5f1c0294286818fe32c699da63b => generated 286 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:49:31.078427 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c503a77e-ca66-4d0b-a8ac-d07f9d37a09c None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:31.179452 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c503a77e-ca66-4d0b-a8ac-d07f9d37a09c None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:31.187782 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:31.212647 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0363db63-0b73-4b23-811e-af1875e4cad7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:31.213727 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0363db63-0b73-4b23-811e-af1875e4cad7 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:31.214006 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0363db63-0b73-4b23-811e-af1875e4cad7 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:31.214272 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0363db63-0b73-4b23-811e-af1875e4cad7 None admin] PATH_INFO: `/v3/projects/041c141bd62d4c238c938ebf575f1653` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:31.215391 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0363db63-0b73-4b23-811e-af1875e4cad7 None admin] RBAC: Authorizing `identity:get_project(project_id=041c141bd62d4c238c938ebf575f1653)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:31.216813 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0363db63-0b73-4b23-811e-af1875e4cad7 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:31.218411 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 418/848] 10.222.0.36 () {56 vars in 1186 bytes} [Thu Oct 20 08:49:31 2022] GET /identity/v3/projects/041c141bd62d4c238c938ebf575f1653 => generated 311 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:49:31.224794 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a74e1e21-fc0b-46e3-bd27-9aa74125a80b None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:31.235235 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:31.321221 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a74e1e21-fc0b-46e3-bd27-9aa74125a80b None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:31.353696 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92e89896-a069-4c2a-a29b-bfe923879f1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:31.354329 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e89896-a069-4c2a-a29b-bfe923879f1b None admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:31.354644 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e89896-a069-4c2a-a29b-bfe923879f1b None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:31.354914 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e89896-a069-4c2a-a29b-bfe923879f1b None admin] PATH_INFO: `/v3/projects/041c141bd62d4c238c938ebf575f1653/users/edf8c5f1c0294286818fe32c699da63b/roles/b3f8d63bf4c4488288e57693c6e9804c` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:31.357004 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92e89896-a069-4c2a-a29b-bfe923879f1b None admin] RBAC: Authorizing `identity:create_grant(project_id=041c141bd62d4c238c938ebf575f1653, user_id=edf8c5f1c0294286818fe32c699da63b, role_id=b3f8d63bf4c4488288e57693c6e9804c)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:31.358426 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92e89896-a069-4c2a-a29b-bfe923879f1b None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:31.372544 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 431/849] 10.222.0.36 () {58 vars in 1439 bytes} [Thu Oct 20 08:49:31 2022] PUT /identity/v3/projects/041c141bd62d4c238c938ebf575f1653/users/edf8c5f1c0294286818fe32c699da63b/roles/b3f8d63bf4c4488288e57693c6e9804c => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 08:49:33.483591 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6791caa-d078-4dfe-a431-2dd012fc7882 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:33.484160 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6791caa-d078-4dfe-a431-2dd012fc7882 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:33.484160 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6791caa-d078-4dfe-a431-2dd012fc7882 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:33.485444 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 419/850] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:49:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:49:33.490675 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca6659b-7753-4a30-830a-4a4d75ebc2c3 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:33.491102 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca6659b-7753-4a30-830a-4a4d75ebc2c3 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:33.491200 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca6659b-7753-4a30-830a-4a4d75ebc2c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:33.509164 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-0ca6659b-7753-4a30-830a-4a4d75ebc2c3 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:49:33.510313 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-0ca6659b-7753-4a30-830a-4a4d75ebc2c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:49:33.573158 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:33.581347 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 432/851] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:49:33 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 20 08:49:33.850415 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6791caa-d078-4dfe-a431-2dd012fc7882 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:33.949105 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6791caa-d078-4dfe-a431-2dd012fc7882 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:33.982801 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47fe2c67-4a27-461b-8174-6137ad2c3ac6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:33.983411 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47fe2c67-4a27-461b-8174-6137ad2c3ac6 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:33.983680 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47fe2c67-4a27-461b-8174-6137ad2c3ac6 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:33.983944 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47fe2c67-4a27-461b-8174-6137ad2c3ac6 None admin] PATH_INFO: `/v3/roles/b3f8d63bf4c4488288e57693c6e9804c` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:33.985307 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47fe2c67-4a27-461b-8174-6137ad2c3ac6 None admin] RBAC: Authorizing `identity:get_role(role_id=b3f8d63bf4c4488288e57693c6e9804c)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:33.986694 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47fe2c67-4a27-461b-8174-6137ad2c3ac6 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:33.987976 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 420/852] 10.222.0.36 () {56 vars in 1177 bytes} [Thu Oct 20 08:49:33 2022] GET /identity/v3/roles/b3f8d63bf4c4488288e57693c6e9804c => generated 223 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:49:33.994229 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca6659b-7753-4a30-830a-4a4d75ebc2c3 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:34.090883 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca6659b-7753-4a30-830a-4a4d75ebc2c3 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:34.122721 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e001c8a5-bcac-438e-9225-61c2bb75e0ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:34.123322 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e001c8a5-bcac-438e-9225-61c2bb75e0ed None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:34.123593 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e001c8a5-bcac-438e-9225-61c2bb75e0ed None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:34.123857 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e001c8a5-bcac-438e-9225-61c2bb75e0ed None admin] PATH_INFO: `/v3/users/edf8c5f1c0294286818fe32c699da63b` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:34.124938 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e001c8a5-bcac-438e-9225-61c2bb75e0ed None admin] RBAC: Authorizing `identity:get_user(user_id=edf8c5f1c0294286818fe32c699da63b)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:34.126341 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e001c8a5-bcac-438e-9225-61c2bb75e0ed None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:34.127883 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 433/853] 10.222.0.36 () {56 vars in 1177 bytes} [Thu Oct 20 08:49:33 2022] GET /identity/v3/users/edf8c5f1c0294286818fe32c699da63b => generated 286 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:49:34.134149 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47fe2c67-4a27-461b-8174-6137ad2c3ac6 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:34.178710 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:34.232727 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47fe2c67-4a27-461b-8174-6137ad2c3ac6 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:34.264572 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07396636-7ab3-4aca-b40b-3a6a3b159d5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:34.265168 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07396636-7ab3-4aca-b40b-3a6a3b159d5a None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:34.265444 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07396636-7ab3-4aca-b40b-3a6a3b159d5a None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:34.265713 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07396636-7ab3-4aca-b40b-3a6a3b159d5a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:34.329513 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07396636-7ab3-4aca-b40b-3a6a3b159d5a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:34.330983 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07396636-7ab3-4aca-b40b-3a6a3b159d5a None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:34.374139 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:34.427999 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 421/854] 10.222.0.36 () {58 vars in 1282 bytes} [Thu Oct 20 08:49:34 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 294 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 20 08:49:34.433956 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e001c8a5-bcac-438e-9225-61c2bb75e0ed None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:34.496964 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:34.530365 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e001c8a5-bcac-438e-9225-61c2bb75e0ed None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:34.562127 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0caf906-5273-468a-b550-dfb38b93ffdd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:34.562756 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0caf906-5273-468a-b550-dfb38b93ffdd None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:34.563030 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0caf906-5273-468a-b550-dfb38b93ffdd None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:34.563293 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0caf906-5273-468a-b550-dfb38b93ffdd None admin] PATH_INFO: `/v3/projects/041c141bd62d4c238c938ebf575f1653` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:34.564619 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0caf906-5273-468a-b550-dfb38b93ffdd None admin] RBAC: Authorizing `identity:get_project(project_id=041c141bd62d4c238c938ebf575f1653)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:34.566027 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0caf906-5273-468a-b550-dfb38b93ffdd None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:34.567615 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 434/855] 10.222.0.36 () {56 vars in 1186 bytes} [Thu Oct 20 08:49:34 2022] GET /identity/v3/projects/041c141bd62d4c238c938ebf575f1653 => generated 311 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:49:34.574466 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07396636-7ab3-4aca-b40b-3a6a3b159d5a None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:34.668780 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07396636-7ab3-4aca-b40b-3a6a3b159d5a None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:34.701531 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9af2f894-c609-40de-bdc3-5c0baab511c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:34.702111 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af2f894-c609-40de-bdc3-5c0baab511c4 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:34.702382 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af2f894-c609-40de-bdc3-5c0baab511c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:34.702673 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af2f894-c609-40de-bdc3-5c0baab511c4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:34.703381 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9af2f894-c609-40de-bdc3-5c0baab511c4 None admin] RBAC: Adding query filter params (role.id=b3f8d63bf4c4488288e57693c6e9804c, scope.project.id=041c141bd62d4c238c938ebf575f1653, user.id=edf8c5f1c0294286818fe32c699da63b) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:49:34.703681 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9af2f894-c609-40de-bdc3-5c0baab511c4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:34.705087 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9af2f894-c609-40de-bdc3-5c0baab511c4 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:34.710954 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 422/856] 10.222.0.36 () {56 vars in 1374 bytes} [Thu Oct 20 08:49:34 2022] GET /identity/v3/role_assignments?user.id=edf8c5f1c0294286818fe32c699da63b&scope.project.id=041c141bd62d4c238c938ebf575f1653&role.id=b3f8d63bf4c4488288e57693c6e9804c => generated 608 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:49:36.817334 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5c16a0-b33f-4057-a6bd-a17d83010e4d None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:36.817839 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5c16a0-b33f-4057-a6bd-a17d83010e4d None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:36.817839 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5c16a0-b33f-4057-a6bd-a17d83010e4d None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:36.819201 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 435/857] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:49:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:49:36.824237 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86bfb491-9ab1-4a06-8a06-c16aadfa964f None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:36.824620 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86bfb491-9ab1-4a06-8a06-c16aadfa964f None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:36.824782 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86bfb491-9ab1-4a06-8a06-c16aadfa964f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:36.843052 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-86bfb491-9ab1-4a06-8a06-c16aadfa964f None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:49:36.844208 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-86bfb491-9ab1-4a06-8a06-c16aadfa964f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:49:36.846079 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:36.907967 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 423/858] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:49:36 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 20 08:49:37.171712 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f5c16a0-b33f-4057-a6bd-a17d83010e4d None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:37.268984 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f5c16a0-b33f-4057-a6bd-a17d83010e4d None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:37.301420 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3641d4e9-38c3-4dd5-8d1f-d8f0e4c025b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:37.302022 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3641d4e9-38c3-4dd5-8d1f-d8f0e4c025b5 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:37.302299 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3641d4e9-38c3-4dd5-8d1f-d8f0e4c025b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:37.302601 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3641d4e9-38c3-4dd5-8d1f-d8f0e4c025b5 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:37.303692 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3641d4e9-38c3-4dd5-8d1f-d8f0e4c025b5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:37.305117 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3641d4e9-38c3-4dd5-8d1f-d8f0e4c025b5 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:37.306659 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 436/859] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:49:37 2022] GET /identity/v3/domains/default => generated 199 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:49:37.312147 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86bfb491-9ab1-4a06-8a06-c16aadfa964f None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:37.407300 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86bfb491-9ab1-4a06-8a06-c16aadfa964f None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:37.438708 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bcd505c-1635-404f-90f7-3ca893376dec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:37.439240 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bcd505c-1635-404f-90f7-3ca893376dec None admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:37.439511 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bcd505c-1635-404f-90f7-3ca893376dec None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:37.439771 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bcd505c-1635-404f-90f7-3ca893376dec None admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:37.440395 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bcd505c-1635-404f-90f7-3ca893376dec None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:37.441813 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bcd505c-1635-404f-90f7-3ca893376dec None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:37.451195 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:37.468476 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 424/860] 10.222.0.36 () {60 vars in 1141 bytes} [Thu Oct 20 08:49:37 2022] POST /identity/v3/projects => generated 311 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 08:49:39.609896 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f316f9a-29fb-4f60-8259-7097a5fa0778 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:39.610456 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f316f9a-29fb-4f60-8259-7097a5fa0778 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:39.610456 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f316f9a-29fb-4f60-8259-7097a5fa0778 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:39.611800 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 437/861] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:49:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:49:39.616894 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f9d6ff-2c1c-488f-a997-4c7aaa926ea6 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:39.617261 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f9d6ff-2c1c-488f-a997-4c7aaa926ea6 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:39.617424 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f9d6ff-2c1c-488f-a997-4c7aaa926ea6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:39.635408 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-43f9d6ff-2c1c-488f-a997-4c7aaa926ea6 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:49:39.636507 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-43f9d6ff-2c1c-488f-a997-4c7aaa926ea6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:49:39.699618 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 425/862] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:49:39 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 20 08:49:39.964231 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f316f9a-29fb-4f60-8259-7097a5fa0778 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:39.967096 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:40.062682 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f316f9a-29fb-4f60-8259-7097a5fa0778 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:40.095792 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4096c722-2b67-467a-ac27-a88561e1b1ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:40.096402 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4096c722-2b67-467a-ac27-a88561e1b1ff None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:40.096679 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4096c722-2b67-467a-ac27-a88561e1b1ff None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:40.096948 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4096c722-2b67-467a-ac27-a88561e1b1ff None admin] PATH_INFO: `/v3/domains/default` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:40.098069 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4096c722-2b67-467a-ac27-a88561e1b1ff None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:40.099527 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4096c722-2b67-467a-ac27-a88561e1b1ff None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:40.101060 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 438/863] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:49:39 2022] GET /identity/v3/domains/default => generated 199 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:49:40.106687 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43f9d6ff-2c1c-488f-a997-4c7aaa926ea6 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:40.201981 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43f9d6ff-2c1c-488f-a997-4c7aaa926ea6 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:40.221252 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:40.235938 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-226cd5ba-b9d6-47ea-8e07-64d106009ea1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:40.236527 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226cd5ba-b9d6-47ea-8e07-64d106009ea1 None admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:40.236803 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226cd5ba-b9d6-47ea-8e07-64d106009ea1 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:40.237071 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226cd5ba-b9d6-47ea-8e07-64d106009ea1 None admin] PATH_INFO: `/v3/users` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:40.237708 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-226cd5ba-b9d6-47ea-8e07-64d106009ea1 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:40.239249 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-226cd5ba-b9d6-47ea-8e07-64d106009ea1 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:40.260000 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 426/864] 10.222.0.36 () {60 vars in 1132 bytes} [Thu Oct 20 08:49:40 2022] POST /identity/v3/users => generated 286 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 08:49:42.416947 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5be3c9-6843-47a9-a137-9ef0185cfd21 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:42.417422 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5be3c9-6843-47a9-a137-9ef0185cfd21 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:42.417522 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5be3c9-6843-47a9-a137-9ef0185cfd21 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:42.418832 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 439/865] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:49:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:49:42.423912 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-809edf47-d3c6-44dd-980f-067b47d689f9 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:42.424302 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-809edf47-d3c6-44dd-980f-067b47d689f9 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:42.424460 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-809edf47-d3c6-44dd-980f-067b47d689f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:42.442283 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-809edf47-d3c6-44dd-980f-067b47d689f9 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:49:42.444034 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-809edf47-d3c6-44dd-980f-067b47d689f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:49:42.508053 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 427/866] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:49:42 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 20 08:49:42.775120 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f5be3c9-6843-47a9-a137-9ef0185cfd21 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:42.843457 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:42.874394 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f5be3c9-6843-47a9-a137-9ef0185cfd21 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:42.907280 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca6db241-f144-4d08-a3de-ccf9333db5e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:42.907882 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6db241-f144-4d08-a3de-ccf9333db5e0 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:42.908154 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6db241-f144-4d08-a3de-ccf9333db5e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:42.908416 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6db241-f144-4d08-a3de-ccf9333db5e0 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:42.918142 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca6db241-f144-4d08-a3de-ccf9333db5e0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 20 08:49:42.922801 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-ca6db241-f144-4d08-a3de-ccf9333db5e0 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 20 08:49:42.922801 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:49:42.922801 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:49:42.922801 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:49:42.922801 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:49:42.922801 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:49:42.922801 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:49:42.922801 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:49:42.922801 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:49:42.922801 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:49:42.922801 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:49:42.922801 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:49:42.922801 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 20 08:49:42.922801 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 20 08:49:42.922801 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 20 08:49:42.922801 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 20 08:49:42.923962 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 20 08:49:42.923962 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 20 08:49:42.923962 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 20 08:49:42.923962 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 20 08:49:42.923962 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:49:42.923962 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:49:42.923962 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:49:42.923962 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:49:42.923962 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:49:42.923962 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:49:42.923962 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:49:42.923962 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:49:42.923962 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:49:42.923962 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:49:42.923962 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:49:42.923962 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:49:42.925048 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:49:42.925048 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:49:42.925048 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:49:42.925048 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:49:42.925048 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 20 08:49:42.925048 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 20 08:49:42.925048 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 20 08:49:42.925048 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 20 08:49:42.925048 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 20 08:49:42.925048 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 20 08:49:42.925048 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 20 08:49:42.925048 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:49:42.925048 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 440/867] 10.222.0.36 () {56 vars in 1096 bytes} [Thu Oct 20 08:49:42 2022] GET /identity/v3/roles/admin => generated 83 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:49:42.930394 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-809edf47-d3c6-44dd-980f-067b47d689f9 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:43.025071 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-809edf47-d3c6-44dd-980f-067b47d689f9 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:43.032113 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:43.059161 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9af5299d-4b98-41aa-bd86-4d3a73aefe49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:43.059743 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af5299d-4b98-41aa-bd86-4d3a73aefe49 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:43.060015 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af5299d-4b98-41aa-bd86-4d3a73aefe49 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:43.060286 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af5299d-4b98-41aa-bd86-4d3a73aefe49 None admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:43.060936 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9af5299d-4b98-41aa-bd86-4d3a73aefe49 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:49:43.061231 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9af5299d-4b98-41aa-bd86-4d3a73aefe49 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:43.062613 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9af5299d-4b98-41aa-bd86-4d3a73aefe49 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:43.071456 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 428/868] 10.222.0.36 () {56 vars in 1099 bytes} [Thu Oct 20 08:49:42 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:49:43.078013 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca6db241-f144-4d08-a3de-ccf9333db5e0 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:43.172081 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca6db241-f144-4d08-a3de-ccf9333db5e0 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:43.198990 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:43.204761 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1ab34a2-e0be-42d8-aaa3-145f8ccd903a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:43.205362 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ab34a2-e0be-42d8-aaa3-145f8ccd903a None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:43.205632 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ab34a2-e0be-42d8-aaa3-145f8ccd903a None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:43.205897 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ab34a2-e0be-42d8-aaa3-145f8ccd903a None admin] PATH_INFO: `/v3/users/7acedc13792144f0874654b3877577e5` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:43.218610 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1ab34a2-e0be-42d8-aaa3-145f8ccd903a None admin] RBAC: Authorizing `identity:get_user(user_id=7acedc13792144f0874654b3877577e5)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:43.220027 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1ab34a2-e0be-42d8-aaa3-145f8ccd903a None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:43.221562 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 441/869] 10.222.0.36 () {56 vars in 1177 bytes} [Thu Oct 20 08:49:43 2022] GET /identity/v3/users/7acedc13792144f0874654b3877577e5 => generated 286 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:49:43.227873 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9af5299d-4b98-41aa-bd86-4d3a73aefe49 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:43.325571 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9af5299d-4b98-41aa-bd86-4d3a73aefe49 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:43.358333 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-336d0493-841d-4dbb-9de6-279ba22cffa5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:43.358972 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-336d0493-841d-4dbb-9de6-279ba22cffa5 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:43.359251 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-336d0493-841d-4dbb-9de6-279ba22cffa5 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:43.359518 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-336d0493-841d-4dbb-9de6-279ba22cffa5 None admin] PATH_INFO: `/v3/projects/0adad275fb9643f9964d88cce4f24d93` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:43.360606 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-336d0493-841d-4dbb-9de6-279ba22cffa5 None admin] RBAC: Authorizing `identity:get_project(project_id=0adad275fb9643f9964d88cce4f24d93)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:43.362024 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-336d0493-841d-4dbb-9de6-279ba22cffa5 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:43.363991 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 429/870] 10.222.0.36 () {56 vars in 1186 bytes} [Thu Oct 20 08:49:43 2022] GET /identity/v3/projects/0adad275fb9643f9964d88cce4f24d93 => generated 311 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:49:43.370230 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1ab34a2-e0be-42d8-aaa3-145f8ccd903a None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:43.463324 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1ab34a2-e0be-42d8-aaa3-145f8ccd903a None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:43.495471 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48caff87-7294-4f96-8c3a-da3a00249e8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:43.496095 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48caff87-7294-4f96-8c3a-da3a00249e8d None admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:43.496365 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48caff87-7294-4f96-8c3a-da3a00249e8d None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:43.496627 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48caff87-7294-4f96-8c3a-da3a00249e8d None admin] PATH_INFO: `/v3/projects/0adad275fb9643f9964d88cce4f24d93/users/7acedc13792144f0874654b3877577e5/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:43.498438 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48caff87-7294-4f96-8c3a-da3a00249e8d None admin] RBAC: Authorizing `identity:create_grant(project_id=0adad275fb9643f9964d88cce4f24d93, user_id=7acedc13792144f0874654b3877577e5, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:43.499862 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48caff87-7294-4f96-8c3a-da3a00249e8d None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:43.511907 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 442/871] 10.222.0.36 () {58 vars in 1439 bytes} [Thu Oct 20 08:49:43 2022] PUT /identity/v3/projects/0adad275fb9643f9964d88cce4f24d93/users/7acedc13792144f0874654b3877577e5/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 142 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 08:49:45.606048 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d97195f5-4222-44ea-bf56-7d0e97e0cffd None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:45.606595 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d97195f5-4222-44ea-bf56-7d0e97e0cffd None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:45.606595 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d97195f5-4222-44ea-bf56-7d0e97e0cffd None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:45.607934 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 430/872] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:49:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:49:45.613080 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4312c4b3-7890-43aa-b97e-1a38ca8bcef2 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:45.613504 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4312c4b3-7890-43aa-b97e-1a38ca8bcef2 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:45.613598 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4312c4b3-7890-43aa-b97e-1a38ca8bcef2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:45.631784 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-4312c4b3-7890-43aa-b97e-1a38ca8bcef2 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:49:45.632880 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-4312c4b3-7890-43aa-b97e-1a38ca8bcef2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:49:45.674308 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:45.696997 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 443/873] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:49:45 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 20 08:49:45.972452 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d97195f5-4222-44ea-bf56-7d0e97e0cffd None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:46.011565 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:46.075076 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d97195f5-4222-44ea-bf56-7d0e97e0cffd None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:46.107403 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f28cc98-0b43-4526-b26d-561169544636 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:46.108010 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f28cc98-0b43-4526-b26d-561169544636 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:46.108282 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f28cc98-0b43-4526-b26d-561169544636 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:46.108547 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f28cc98-0b43-4526-b26d-561169544636 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:46.118884 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f28cc98-0b43-4526-b26d-561169544636 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 20 08:49:46.123501 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-0f28cc98-0b43-4526-b26d-561169544636 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 20 08:49:46.123501 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:49:46.123501 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:49:46.123501 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:49:46.123501 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:49:46.123501 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:49:46.123501 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:49:46.123501 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:49:46.123501 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:49:46.123501 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:49:46.123501 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:49:46.123501 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:49:46.123501 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 20 08:49:46.123501 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 20 08:49:46.123501 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 20 08:49:46.123501 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 20 08:49:46.124569 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 20 08:49:46.124569 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 20 08:49:46.124569 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 20 08:49:46.124569 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 20 08:49:46.124569 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:49:46.124569 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:49:46.124569 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:49:46.124569 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:49:46.124569 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:49:46.124569 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:49:46.124569 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:49:46.124569 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:49:46.124569 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:49:46.124569 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:49:46.124569 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:49:46.124569 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:49:46.125464 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:49:46.125464 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:49:46.125464 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:49:46.125464 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:49:46.125464 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 20 08:49:46.125464 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 20 08:49:46.125464 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 20 08:49:46.125464 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 20 08:49:46.125464 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 20 08:49:46.125464 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 20 08:49:46.125464 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 20 08:49:46.125464 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:49:46.125464 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 431/874] 10.222.0.36 () {56 vars in 1096 bytes} [Thu Oct 20 08:49:45 2022] GET /identity/v3/roles/admin => generated 83 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:49:46.131190 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4312c4b3-7890-43aa-b97e-1a38ca8bcef2 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:46.223914 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4312c4b3-7890-43aa-b97e-1a38ca8bcef2 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:46.256374 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8850bf0-36ea-4aab-b231-0aa8274788a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:46.256959 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8850bf0-36ea-4aab-b231-0aa8274788a6 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:46.257232 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8850bf0-36ea-4aab-b231-0aa8274788a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:46.257501 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8850bf0-36ea-4aab-b231-0aa8274788a6 None admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:46.258144 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8850bf0-36ea-4aab-b231-0aa8274788a6 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:49:46.258436 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8850bf0-36ea-4aab-b231-0aa8274788a6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:46.259806 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8850bf0-36ea-4aab-b231-0aa8274788a6 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:46.267611 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 444/875] 10.222.0.36 () {56 vars in 1099 bytes} [Thu Oct 20 08:49:46 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:49:46.273673 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f28cc98-0b43-4526-b26d-561169544636 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:46.364328 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:46.371433 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f28cc98-0b43-4526-b26d-561169544636 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:46.404417 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6765cc57-ed7b-439d-b9ce-788a05e17d26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:46.405038 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6765cc57-ed7b-439d-b9ce-788a05e17d26 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:46.405313 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6765cc57-ed7b-439d-b9ce-788a05e17d26 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:46.405580 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6765cc57-ed7b-439d-b9ce-788a05e17d26 None admin] PATH_INFO: `/v3/users/7acedc13792144f0874654b3877577e5` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:46.406699 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6765cc57-ed7b-439d-b9ce-788a05e17d26 None admin] RBAC: Authorizing `identity:get_user(user_id=7acedc13792144f0874654b3877577e5)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:46.408167 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6765cc57-ed7b-439d-b9ce-788a05e17d26 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:46.409717 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 432/876] 10.222.0.36 () {56 vars in 1177 bytes} [Thu Oct 20 08:49:46 2022] GET /identity/v3/users/7acedc13792144f0874654b3877577e5 => generated 286 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:49:46.416073 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8850bf0-36ea-4aab-b231-0aa8274788a6 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:46.457358 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:46.514528 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8850bf0-36ea-4aab-b231-0aa8274788a6 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:46.546967 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54cc0bd3-cc52-4bce-a4c9-ed333dd8a24e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:46.547559 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54cc0bd3-cc52-4bce-a4c9-ed333dd8a24e None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:46.547835 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54cc0bd3-cc52-4bce-a4c9-ed333dd8a24e None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:46.548102 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54cc0bd3-cc52-4bce-a4c9-ed333dd8a24e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:46.612247 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54cc0bd3-cc52-4bce-a4c9-ed333dd8a24e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:46.613673 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54cc0bd3-cc52-4bce-a4c9-ed333dd8a24e None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:46.651569 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:46.706843 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 445/877] 10.222.0.36 () {58 vars in 1282 bytes} [Thu Oct 20 08:49:46 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 291 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 20 08:49:46.713618 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6765cc57-ed7b-439d-b9ce-788a05e17d26 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:46.809682 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6765cc57-ed7b-439d-b9ce-788a05e17d26 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:46.841613 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83149672-cd19-45e4-a1d7-63da1f3a8a43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:46.842220 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83149672-cd19-45e4-a1d7-63da1f3a8a43 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:46.842509 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83149672-cd19-45e4-a1d7-63da1f3a8a43 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:46.842778 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83149672-cd19-45e4-a1d7-63da1f3a8a43 None admin] PATH_INFO: `/v3/projects/0adad275fb9643f9964d88cce4f24d93` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:46.843852 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83149672-cd19-45e4-a1d7-63da1f3a8a43 None admin] RBAC: Authorizing `identity:get_project(project_id=0adad275fb9643f9964d88cce4f24d93)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:46.845269 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83149672-cd19-45e4-a1d7-63da1f3a8a43 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:46.846873 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 433/878] 10.222.0.36 () {56 vars in 1186 bytes} [Thu Oct 20 08:49:46 2022] GET /identity/v3/projects/0adad275fb9643f9964d88cce4f24d93 => generated 311 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:49:46.853355 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54cc0bd3-cc52-4bce-a4c9-ed333dd8a24e None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:46.947672 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54cc0bd3-cc52-4bce-a4c9-ed333dd8a24e None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:46.980410 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1112a409-6490-4329-a7af-f7fabec07196 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:46.980994 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1112a409-6490-4329-a7af-f7fabec07196 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:46.981271 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1112a409-6490-4329-a7af-f7fabec07196 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:46.981538 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1112a409-6490-4329-a7af-f7fabec07196 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:46.982252 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1112a409-6490-4329-a7af-f7fabec07196 None admin] RBAC: Adding query filter params (role.id=d9d3bc62f6e8453488c74856ef333bb3, scope.project.id=0adad275fb9643f9964d88cce4f24d93, user.id=7acedc13792144f0874654b3877577e5) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:49:46.982582 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1112a409-6490-4329-a7af-f7fabec07196 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:46.983992 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1112a409-6490-4329-a7af-f7fabec07196 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:46.989552 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 446/879] 10.222.0.36 () {56 vars in 1374 bytes} [Thu Oct 20 08:49:46 2022] GET /identity/v3/role_assignments?user.id=7acedc13792144f0874654b3877577e5&scope.project.id=0adad275fb9643f9964d88cce4f24d93&role.id=d9d3bc62f6e8453488c74856ef333bb3 => generated 608 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:49:49.109689 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69672efc-2d42-413d-94b1-cf6bb4e6bc10 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:49.110241 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69672efc-2d42-413d-94b1-cf6bb4e6bc10 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:49.110241 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69672efc-2d42-413d-94b1-cf6bb4e6bc10 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:49.111574 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 434/880] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:49:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:49:49.117251 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-086602ac-c0bc-4bdb-8d9b-19948e5be4dd None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:49.117648 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-086602ac-c0bc-4bdb-8d9b-19948e5be4dd None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:49.117798 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-086602ac-c0bc-4bdb-8d9b-19948e5be4dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:49.124992 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:49.136828 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-086602ac-c0bc-4bdb-8d9b-19948e5be4dd None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:49:49.138008 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-086602ac-c0bc-4bdb-8d9b-19948e5be4dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:49:49.201836 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 447/881] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:49:49 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 20 08:49:49.467846 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69672efc-2d42-413d-94b1-cf6bb4e6bc10 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:49.498987 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:49.566048 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69672efc-2d42-413d-94b1-cf6bb4e6bc10 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:49.598300 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeca5da6-dfe2-4aaa-91cc-282d30de28e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:49.598926 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeca5da6-dfe2-4aaa-91cc-282d30de28e0 None admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:49.599815 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeca5da6-dfe2-4aaa-91cc-282d30de28e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:49.600126 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeca5da6-dfe2-4aaa-91cc-282d30de28e0 None admin] PATH_INFO: `/v3/domains` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:49.600783 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeca5da6-dfe2-4aaa-91cc-282d30de28e0 None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:49.602127 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeca5da6-dfe2-4aaa-91cc-282d30de28e0 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:49.616821 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 435/882] 10.222.0.36 () {60 vars in 1138 bytes} [Thu Oct 20 08:49:49 2022] POST /identity/v3/domains => generated 267 bytes in 149 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 08:49:51.756366 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ebc180-85aa-4b4b-afe5-2dd2a9b96462 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:51.756911 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ebc180-85aa-4b4b-afe5-2dd2a9b96462 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:51.756911 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ebc180-85aa-4b4b-afe5-2dd2a9b96462 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:51.758159 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 448/883] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:49:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:49:51.763549 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611f5d67-3a44-4b00-b591-c333f25d62a8 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:51.763946 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611f5d67-3a44-4b00-b591-c333f25d62a8 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:51.764088 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611f5d67-3a44-4b00-b591-c333f25d62a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:51.782750 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-611f5d67-3a44-4b00-b591-c333f25d62a8 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:49:51.783913 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-611f5d67-3a44-4b00-b591-c333f25d62a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:49:51.847015 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 436/884] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:49:51 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 20 08:49:52.112978 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0ebc180-85aa-4b4b-afe5-2dd2a9b96462 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:52.211567 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0ebc180-85aa-4b4b-afe5-2dd2a9b96462 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:52.243159 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49e0fc7e-5798-49f7-b884-e7ca16f8a251 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:52.243760 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e0fc7e-5798-49f7-b884-e7ca16f8a251 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:52.244038 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e0fc7e-5798-49f7-b884-e7ca16f8a251 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:52.244306 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e0fc7e-5798-49f7-b884-e7ca16f8a251 None admin] PATH_INFO: `/v3/domains/7406fd488df242a092c4a16462c0d432` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:52.246145 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:52.255812 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49e0fc7e-5798-49f7-b884-e7ca16f8a251 None admin] RBAC: Authorizing `identity:get_domain(domain_id=7406fd488df242a092c4a16462c0d432)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:52.257238 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49e0fc7e-5798-49f7-b884-e7ca16f8a251 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:52.258823 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 449/885] 10.222.0.36 () {56 vars in 1183 bytes} [Thu Oct 20 08:49:52 2022] GET /identity/v3/domains/7406fd488df242a092c4a16462c0d432 => generated 267 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:49:52.264536 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-611f5d67-3a44-4b00-b591-c333f25d62a8 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:52.287948 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:52.360904 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-611f5d67-3a44-4b00-b591-c333f25d62a8 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:52.393968 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3186993-3420-4487-af47-5872875f7141 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:52.394656 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3186993-3420-4487-af47-5872875f7141 None admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:52.394933 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3186993-3420-4487-af47-5872875f7141 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:52.395198 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3186993-3420-4487-af47-5872875f7141 None admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:52.395841 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3186993-3420-4487-af47-5872875f7141 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:52.397873 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3186993-3420-4487-af47-5872875f7141 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:52.423469 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 437/886] 10.222.0.36 () {60 vars in 1141 bytes} [Thu Oct 20 08:49:52 2022] POST /identity/v3/projects => generated 361 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 08:49:54.559491 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce53e7f8-d225-4dfa-8d35-4c51c922943c None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:54.560031 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce53e7f8-d225-4dfa-8d35-4c51c922943c None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:54.560031 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce53e7f8-d225-4dfa-8d35-4c51c922943c None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:54.561369 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 450/887] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:49:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:49:54.566548 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79fb5d62-3a91-44d7-ad0a-faa3710d61f3 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:54.567004 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79fb5d62-3a91-44d7-ad0a-faa3710d61f3 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:54.567103 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79fb5d62-3a91-44d7-ad0a-faa3710d61f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:54.584751 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-79fb5d62-3a91-44d7-ad0a-faa3710d61f3 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:49:54.586171 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-79fb5d62-3a91-44d7-ad0a-faa3710d61f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:49:54.649351 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 438/888] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:49:54 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 20 08:49:54.918970 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce53e7f8-d225-4dfa-8d35-4c51c922943c None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:55.017087 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce53e7f8-d225-4dfa-8d35-4c51c922943c None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:55.050023 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea916af9-e353-4b14-b1cd-8369af753d5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:55.050653 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea916af9-e353-4b14-b1cd-8369af753d5b None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:55.050933 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea916af9-e353-4b14-b1cd-8369af753d5b None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:55.051202 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea916af9-e353-4b14-b1cd-8369af753d5b None admin] PATH_INFO: `/v3/domains/7406fd488df242a092c4a16462c0d432` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:55.052294 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea916af9-e353-4b14-b1cd-8369af753d5b None admin] RBAC: Authorizing `identity:get_domain(domain_id=7406fd488df242a092c4a16462c0d432)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:55.053749 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea916af9-e353-4b14-b1cd-8369af753d5b None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:55.055288 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 451/889] 10.222.0.36 () {56 vars in 1183 bytes} [Thu Oct 20 08:49:54 2022] GET /identity/v3/domains/7406fd488df242a092c4a16462c0d432 => generated 267 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:49:55.060795 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79fb5d62-3a91-44d7-ad0a-faa3710d61f3 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:55.063836 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:55.159968 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79fb5d62-3a91-44d7-ad0a-faa3710d61f3 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:55.191925 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf2b07db-dabc-4092-ab0d-e6c6447133d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:55.192463 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf2b07db-dabc-4092-ab0d-e6c6447133d7 None admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:55.192740 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf2b07db-dabc-4092-ab0d-e6c6447133d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:55.193011 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf2b07db-dabc-4092-ab0d-e6c6447133d7 None admin] PATH_INFO: `/v3/users` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:55.194170 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf2b07db-dabc-4092-ab0d-e6c6447133d7 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:55.195634 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf2b07db-dabc-4092-ab0d-e6c6447133d7 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:55.216601 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 439/890] 10.222.0.36 () {60 vars in 1132 bytes} [Thu Oct 20 08:49:55 2022] POST /identity/v3/users => generated 311 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 08:49:57.370001 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb018660-dace-4f01-83b2-3459330c501c None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:57.370548 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb018660-dace-4f01-83b2-3459330c501c None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:57.370548 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb018660-dace-4f01-83b2-3459330c501c None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:57.371898 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 452/891] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:49:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:49:57.377078 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39444f55-e84e-4108-8abe-0f32d4efdc6f None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:57.377514 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39444f55-e84e-4108-8abe-0f32d4efdc6f None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:57.377610 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39444f55-e84e-4108-8abe-0f32d4efdc6f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:57.395929 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-39444f55-e84e-4108-8abe-0f32d4efdc6f None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:49:57.397086 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-39444f55-e84e-4108-8abe-0f32d4efdc6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:49:57.449651 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:57.461855 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 440/892] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:49:57 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 20 08:49:57.727232 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb018660-dace-4f01-83b2-3459330c501c None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:57.786005 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:57.823604 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb018660-dace-4f01-83b2-3459330c501c None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:57.856164 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbdb29c0-e787-4207-9c1e-ee055d19f342 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:57.856767 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbdb29c0-e787-4207-9c1e-ee055d19f342 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:57.857041 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbdb29c0-e787-4207-9c1e-ee055d19f342 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:57.857307 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbdb29c0-e787-4207-9c1e-ee055d19f342 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:57.867107 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbdb29c0-e787-4207-9c1e-ee055d19f342 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 20 08:49:57.871732 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-bbdb29c0-e787-4207-9c1e-ee055d19f342 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 20 08:49:57.871732 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:49:57.871732 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:49:57.871732 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:49:57.871732 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:49:57.871732 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:49:57.871732 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:49:57.871732 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:49:57.871732 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:49:57.871732 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:49:57.871732 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:49:57.871732 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:49:57.871732 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 20 08:49:57.871732 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 20 08:49:57.871732 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 20 08:49:57.871732 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 20 08:49:57.872883 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 20 08:49:57.872883 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 20 08:49:57.872883 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 20 08:49:57.872883 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 20 08:49:57.872883 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:49:57.872883 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:49:57.872883 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:49:57.872883 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:49:57.872883 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:49:57.872883 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:49:57.872883 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:49:57.872883 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:49:57.872883 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:49:57.872883 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:49:57.872883 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:49:57.872883 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:49:57.873843 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:49:57.873843 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:49:57.873843 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:49:57.873843 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:49:57.873843 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 20 08:49:57.873843 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 20 08:49:57.873843 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 20 08:49:57.873843 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 20 08:49:57.873843 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 20 08:49:57.873843 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 20 08:49:57.873843 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 20 08:49:57.873843 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:49:57.873843 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 453/893] 10.222.0.36 () {56 vars in 1096 bytes} [Thu Oct 20 08:49:57 2022] GET /identity/v3/roles/admin => generated 83 bytes in 146 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:49:57.878716 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39444f55-e84e-4108-8abe-0f32d4efdc6f None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:57.973192 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39444f55-e84e-4108-8abe-0f32d4efdc6f None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:58.006505 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b631c760-85e7-4726-9ab5-f9677c63431c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:58.007029 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b631c760-85e7-4726-9ab5-f9677c63431c None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:58.007300 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b631c760-85e7-4726-9ab5-f9677c63431c None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:58.007563 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b631c760-85e7-4726-9ab5-f9677c63431c None admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:58.008214 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b631c760-85e7-4726-9ab5-f9677c63431c None admin] RBAC: Adding query filter params (name=admin) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:49:58.008508 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b631c760-85e7-4726-9ab5-f9677c63431c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:58.009870 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b631c760-85e7-4726-9ab5-f9677c63431c None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:58.017489 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 441/894] 10.222.0.36 () {56 vars in 1099 bytes} [Thu Oct 20 08:49:57 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:49:58.023629 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbdb29c0-e787-4207-9c1e-ee055d19f342 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:58.118537 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbdb29c0-e787-4207-9c1e-ee055d19f342 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:58.140237 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:58.151933 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb5699ba-0f6b-4d43-a7b1-3bb8b52b5ed7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:58.152540 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb5699ba-0f6b-4d43-a7b1-3bb8b52b5ed7 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:58.152812 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb5699ba-0f6b-4d43-a7b1-3bb8b52b5ed7 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:58.153077 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb5699ba-0f6b-4d43-a7b1-3bb8b52b5ed7 None admin] PATH_INFO: `/v3/users/8c1ebfda773a48f99878bf6499fccc0f` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:58.165527 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb5699ba-0f6b-4d43-a7b1-3bb8b52b5ed7 None admin] RBAC: Authorizing `identity:get_user(user_id=8c1ebfda773a48f99878bf6499fccc0f)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:58.166992 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb5699ba-0f6b-4d43-a7b1-3bb8b52b5ed7 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:58.168533 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 454/895] 10.222.0.36 () {56 vars in 1177 bytes} [Thu Oct 20 08:49:58 2022] GET /identity/v3/users/8c1ebfda773a48f99878bf6499fccc0f => generated 311 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:49:58.174283 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b631c760-85e7-4726-9ab5-f9677c63431c None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:58.225287 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:49:58.271442 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b631c760-85e7-4726-9ab5-f9677c63431c None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:58.303685 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0250340-aa5c-4257-a93f-9d754b377872 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:58.304299 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0250340-aa5c-4257-a93f-9d754b377872 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:58.304574 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0250340-aa5c-4257-a93f-9d754b377872 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:58.304842 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0250340-aa5c-4257-a93f-9d754b377872 None admin] PATH_INFO: `/v3/projects/eb8565091d5f4f37a691f715224d0b28` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:58.305931 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0250340-aa5c-4257-a93f-9d754b377872 None admin] RBAC: Authorizing `identity:get_project(project_id=eb8565091d5f4f37a691f715224d0b28)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:58.307384 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0250340-aa5c-4257-a93f-9d754b377872 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:58.308993 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 442/896] 10.222.0.36 () {56 vars in 1186 bytes} [Thu Oct 20 08:49:58 2022] GET /identity/v3/projects/eb8565091d5f4f37a691f715224d0b28 => generated 361 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:49:58.314965 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb5699ba-0f6b-4d43-a7b1-3bb8b52b5ed7 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:49:58.411723 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb5699ba-0f6b-4d43-a7b1-3bb8b52b5ed7 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:49:58.444105 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-849c5960-70b7-4c40-b9e4-3975586c4ac5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:49:58.444752 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-849c5960-70b7-4c40-b9e4-3975586c4ac5 None admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:49:58.445028 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-849c5960-70b7-4c40-b9e4-3975586c4ac5 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:49:58.445299 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-849c5960-70b7-4c40-b9e4-3975586c4ac5 None admin] PATH_INFO: `/v3/projects/eb8565091d5f4f37a691f715224d0b28/users/8c1ebfda773a48f99878bf6499fccc0f/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:49:58.447171 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-849c5960-70b7-4c40-b9e4-3975586c4ac5 None admin] RBAC: Authorizing `identity:create_grant(project_id=eb8565091d5f4f37a691f715224d0b28, user_id=8c1ebfda773a48f99878bf6499fccc0f, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:49:58.448684 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-849c5960-70b7-4c40-b9e4-3975586c4ac5 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:49:58.461579 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 455/897] 10.222.0.36 () {58 vars in 1439 bytes} [Thu Oct 20 08:49:58 2022] PUT /identity/v3/projects/eb8565091d5f4f37a691f715224d0b28/users/8c1ebfda773a48f99878bf6499fccc0f/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 08:50:00.540018 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de6d6995-c12c-4ec6-802e-9a66dd3ef4a9 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:50:00.540542 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de6d6995-c12c-4ec6-802e-9a66dd3ef4a9 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:50:00.540542 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de6d6995-c12c-4ec6-802e-9a66dd3ef4a9 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:50:00.541872 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 443/898] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:50:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:50:00.547012 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-208c05b7-f418-4834-a359-81efb4846168 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:50:00.547461 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-208c05b7-f418-4834-a359-81efb4846168 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:50:00.547560 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-208c05b7-f418-4834-a359-81efb4846168 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:50:00.565784 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-208c05b7-f418-4834-a359-81efb4846168 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:50:00.566998 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-208c05b7-f418-4834-a359-81efb4846168 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:50:00.600076 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:50:00.632124 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 456/899] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:50:00 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 20 08:50:00.897962 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de6d6995-c12c-4ec6-802e-9a66dd3ef4a9 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:50:00.994942 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de6d6995-c12c-4ec6-802e-9a66dd3ef4a9 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:50:01.013716 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:50:01.027699 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a681c11a-c802-4723-8f6c-1eb209acff0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:50:01.028300 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a681c11a-c802-4723-8f6c-1eb209acff0b None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:50:01.028570 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a681c11a-c802-4723-8f6c-1eb209acff0b None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:50:01.028833 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a681c11a-c802-4723-8f6c-1eb209acff0b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:50:01.039146 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a681c11a-c802-4723-8f6c-1eb209acff0b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 20 08:50:01.043798 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-a681c11a-c802-4723-8f6c-1eb209acff0b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 20 08:50:01.043798 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:50:01.043798 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:50:01.043798 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:50:01.043798 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:50:01.043798 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:50:01.043798 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:50:01.043798 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:50:01.043798 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:50:01.043798 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:50:01.043798 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:50:01.043798 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:50:01.043798 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 20 08:50:01.043798 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 20 08:50:01.043798 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 20 08:50:01.043798 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 20 08:50:01.044918 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 20 08:50:01.044918 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 20 08:50:01.044918 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 20 08:50:01.044918 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 20 08:50:01.044918 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:50:01.044918 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:50:01.044918 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:50:01.044918 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:50:01.044918 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:50:01.044918 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:50:01.044918 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:50:01.044918 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:50:01.044918 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:50:01.044918 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:50:01.044918 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:50:01.044918 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:50:01.045993 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:50:01.045993 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:50:01.045993 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:50:01.045993 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:50:01.045993 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 20 08:50:01.045993 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 20 08:50:01.045993 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 20 08:50:01.045993 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 20 08:50:01.045993 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 20 08:50:01.045993 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 20 08:50:01.045993 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 20 08:50:01.045993 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:50:01.045993 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 444/900] 10.222.0.36 () {56 vars in 1096 bytes} [Thu Oct 20 08:50:00 2022] GET /identity/v3/roles/admin => generated 83 bytes in 147 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:50:01.050767 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-208c05b7-f418-4834-a359-81efb4846168 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:50:01.148510 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-208c05b7-f418-4834-a359-81efb4846168 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:50:01.180314 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac2d58d5-3dd6-453b-954a-71500e97864c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:50:01.180899 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2d58d5-3dd6-453b-954a-71500e97864c None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:50:01.181171 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2d58d5-3dd6-453b-954a-71500e97864c None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:50:01.181432 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2d58d5-3dd6-453b-954a-71500e97864c None admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:50:01.182079 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac2d58d5-3dd6-453b-954a-71500e97864c None admin] RBAC: Adding query filter params (name=admin) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:50:01.182372 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac2d58d5-3dd6-453b-954a-71500e97864c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:50:01.183752 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac2d58d5-3dd6-453b-954a-71500e97864c None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:50:01.191657 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 457/901] 10.222.0.36 () {56 vars in 1099 bytes} [Thu Oct 20 08:50:01 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 08:50:01.198800 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a681c11a-c802-4723-8f6c-1eb209acff0b None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:50:01.294050 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a681c11a-c802-4723-8f6c-1eb209acff0b None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:50:01.326337 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4758bd58-c6fd-485e-b109-4e86bdf00593 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:50:01.326968 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4758bd58-c6fd-485e-b109-4e86bdf00593 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:50:01.327248 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4758bd58-c6fd-485e-b109-4e86bdf00593 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:50:01.327513 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4758bd58-c6fd-485e-b109-4e86bdf00593 None admin] PATH_INFO: `/v3/users/8c1ebfda773a48f99878bf6499fccc0f` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:50:01.328595 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4758bd58-c6fd-485e-b109-4e86bdf00593 None admin] RBAC: Authorizing `identity:get_user(user_id=8c1ebfda773a48f99878bf6499fccc0f)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:50:01.330107 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4758bd58-c6fd-485e-b109-4e86bdf00593 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:50:01.331674 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 445/902] 10.222.0.36 () {56 vars in 1177 bytes} [Thu Oct 20 08:50:01 2022] GET /identity/v3/users/8c1ebfda773a48f99878bf6499fccc0f => generated 311 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:50:01.338079 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac2d58d5-3dd6-453b-954a-71500e97864c None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:50:01.368040 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:50:01.434966 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac2d58d5-3dd6-453b-954a-71500e97864c None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:50:01.468104 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa91a83a-80b8-47c5-b9f3-c3f538594e5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:50:01.468695 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa91a83a-80b8-47c5-b9f3-c3f538594e5e None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:50:01.468971 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa91a83a-80b8-47c5-b9f3-c3f538594e5e None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:50:01.469239 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa91a83a-80b8-47c5-b9f3-c3f538594e5e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:50:01.533066 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa91a83a-80b8-47c5-b9f3-c3f538594e5e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:50:01.534502 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa91a83a-80b8-47c5-b9f3-c3f538594e5e None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:50:01.562315 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:50:01.627199 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 458/903] 10.222.0.36 () {58 vars in 1282 bytes} [Thu Oct 20 08:50:01 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 290 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 20 08:50:01.633651 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4758bd58-c6fd-485e-b109-4e86bdf00593 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:50:01.680389 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:50:01.729179 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4758bd58-c6fd-485e-b109-4e86bdf00593 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:50:01.761541 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a65ed07-95e7-491c-965b-78f4098ebcf0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:50:01.762139 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a65ed07-95e7-491c-965b-78f4098ebcf0 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:50:01.762411 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a65ed07-95e7-491c-965b-78f4098ebcf0 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:50:01.762701 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a65ed07-95e7-491c-965b-78f4098ebcf0 None admin] PATH_INFO: `/v3/projects/eb8565091d5f4f37a691f715224d0b28` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:50:01.763773 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a65ed07-95e7-491c-965b-78f4098ebcf0 None admin] RBAC: Authorizing `identity:get_project(project_id=eb8565091d5f4f37a691f715224d0b28)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:50:01.765191 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a65ed07-95e7-491c-965b-78f4098ebcf0 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:50:01.766796 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 446/904] 10.222.0.36 () {56 vars in 1186 bytes} [Thu Oct 20 08:50:01 2022] GET /identity/v3/projects/eb8565091d5f4f37a691f715224d0b28 => generated 361 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:50:01.772946 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa91a83a-80b8-47c5-b9f3-c3f538594e5e None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:50:01.868866 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa91a83a-80b8-47c5-b9f3-c3f538594e5e None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:50:01.896616 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:50:01.901477 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5109ecf8-5d57-40e2-8425-ec20498e33da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:50:01.902057 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5109ecf8-5d57-40e2-8425-ec20498e33da None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:50:01.902328 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5109ecf8-5d57-40e2-8425-ec20498e33da None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:50:01.902616 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5109ecf8-5d57-40e2-8425-ec20498e33da None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:50:01.903320 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5109ecf8-5d57-40e2-8425-ec20498e33da None admin] RBAC: Adding query filter params (role.id=d9d3bc62f6e8453488c74856ef333bb3, scope.project.id=eb8565091d5f4f37a691f715224d0b28, user.id=8c1ebfda773a48f99878bf6499fccc0f) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:50:01.903617 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5109ecf8-5d57-40e2-8425-ec20498e33da None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:50:01.905019 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5109ecf8-5d57-40e2-8425-ec20498e33da None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:50:01.910545 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 459/905] 10.222.0.36 () {56 vars in 1374 bytes} [Thu Oct 20 08:50:01 2022] GET /identity/v3/role_assignments?user.id=8c1ebfda773a48f99878bf6499fccc0f&scope.project.id=eb8565091d5f4f37a691f715224d0b28&role.id=d9d3bc62f6e8453488c74856ef333bb3 => generated 608 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:50:11.757601 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=42 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:50:11.908071 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=2 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:50:56.998715 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d0ab2c-c8f8-4fd9-91ab-c04475b71460 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:50:56.998715 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d0ab2c-c8f8-4fd9-91ab-c04475b71460 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:50:56.998715 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d0ab2c-c8f8-4fd9-91ab-c04475b71460 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:50:56.998715 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 447/906] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:50:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:50:57.005697 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bbaaa29-0bd9-4ef3-a4a6-5ba36ce160f6 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:50:57.006153 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bbaaa29-0bd9-4ef3-a4a6-5ba36ce160f6 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:50:57.006308 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bbaaa29-0bd9-4ef3-a4a6-5ba36ce160f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:50:57.026163 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-7bbaaa29-0bd9-4ef3-a4a6-5ba36ce160f6 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:50:57.027644 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-7bbaaa29-0bd9-4ef3-a4a6-5ba36ce160f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:50:57.115290 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 460/907] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:50:57 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 20 08:50:57.404347 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34d0ab2c-c8f8-4fd9-91ab-c04475b71460 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:50:57.533947 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34d0ab2c-c8f8-4fd9-91ab-c04475b71460 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:50:57.573286 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-368ab280-46df-4382-9d69-1c296bef9d76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:50:57.574117 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368ab280-46df-4382-9d69-1c296bef9d76 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:50:57.574453 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368ab280-46df-4382-9d69-1c296bef9d76 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:50:57.574748 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368ab280-46df-4382-9d69-1c296bef9d76 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:50:57.587933 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-368ab280-46df-4382-9d69-1c296bef9d76 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:50:57.589678 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-368ab280-46df-4382-9d69-1c296bef9d76 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:50:57.606616 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-368ab280-46df-4382-9d69-1c296bef9d76 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:50:57.606616 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:50:57.606616 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:50:57.606616 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:50:57.606616 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:50:57.606616 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:50:57.606616 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:50:57.606616 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:50:57.606616 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:50:57.606616 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:50:57.606616 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:50:57.606616 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:50:57.606616 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 20 08:50:57.606616 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 20 08:50:57.606616 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 20 08:50:57.606616 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 20 08:50:57.610824 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:50:57.610824 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:50:57.610824 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:50:57.610824 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:50:57.610824 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:50:57.610824 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:50:57.610824 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:50:57.610824 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:50:57.610824 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:50:57.610824 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:50:57.610824 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:50:57.610824 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:50:57.610824 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:50:57.610824 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:50:57.610824 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:50:57.610824 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:50:57.610824 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 20 08:50:57.612070 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 20 08:50:57.612070 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:50:57.612070 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:50:57.612070 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 448/908] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:50:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:50:57.616835 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bbaaa29-0bd9-4ef3-a4a6-5ba36ce160f6 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:50:57.741663 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bbaaa29-0bd9-4ef3-a4a6-5ba36ce160f6 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:50:57.775709 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:50:57.788222 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9083d360-62c0-4985-8295-67bbc9de98de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:50:57.788840 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9083d360-62c0-4985-8295-67bbc9de98de None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:50:57.789165 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9083d360-62c0-4985-8295-67bbc9de98de None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:50:57.789472 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9083d360-62c0-4985-8295-67bbc9de98de None admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:50:57.790150 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9083d360-62c0-4985-8295-67bbc9de98de None admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:50:57.790496 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9083d360-62c0-4985-8295-67bbc9de98de None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:50:57.791987 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9083d360-62c0-4985-8295-67bbc9de98de None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:50:57.807164 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 461/909] 10.222.0.36 () {56 vars in 1109 bytes} [Thu Oct 20 08:50:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:50:57.813260 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-368ab280-46df-4382-9d69-1c296bef9d76 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:50:57.893524 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:50:57.959235 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-368ab280-46df-4382-9d69-1c296bef9d76 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:50:58.016100 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff1d1442-2100-4af9-8759-f6802b394ade None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:50:58.016749 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff1d1442-2100-4af9-8759-f6802b394ade None admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:50:58.016931 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff1d1442-2100-4af9-8759-f6802b394ade None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:50:58.017204 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff1d1442-2100-4af9-8759-f6802b394ade None admin] PATH_INFO: `/v3/users` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:50:58.017861 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff1d1442-2100-4af9-8759-f6802b394ade None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:50:58.019451 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff1d1442-2100-4af9-8759-f6802b394ade None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:50:58.045192 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 449/910] 10.222.0.36 () {60 vars in 1132 bytes} [Thu Oct 20 08:50:57 2022] POST /identity/v3/users => generated 251 bytes in 232 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 08:51:00.281388 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-383feeab-2d5d-4a8f-b966-a3f154370324 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:00.281943 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-383feeab-2d5d-4a8f-b966-a3f154370324 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:00.281943 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-383feeab-2d5d-4a8f-b966-a3f154370324 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:00.283288 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 462/911] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:51:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:51:00.289241 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc73fadd-43b5-4908-9a8e-8d25989ea1c8 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:00.289691 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc73fadd-43b5-4908-9a8e-8d25989ea1c8 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:00.289789 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc73fadd-43b5-4908-9a8e-8d25989ea1c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:00.308689 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-cc73fadd-43b5-4908-9a8e-8d25989ea1c8 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:51:00.309889 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-cc73fadd-43b5-4908-9a8e-8d25989ea1c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:51:00.398508 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 450/912] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:51:00 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 20 08:51:00.690135 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-383feeab-2d5d-4a8f-b966-a3f154370324 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:51:00.807937 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-383feeab-2d5d-4a8f-b966-a3f154370324 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:51:00.843623 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8138992f-6a75-47cd-9ef8-48f116ead455 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:51:00.844210 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8138992f-6a75-47cd-9ef8-48f116ead455 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:00.844489 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8138992f-6a75-47cd-9ef8-48f116ead455 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:00.844753 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8138992f-6a75-47cd-9ef8-48f116ead455 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:00.856665 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8138992f-6a75-47cd-9ef8-48f116ead455 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 20 08:51:00.861411 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-8138992f-6a75-47cd-9ef8-48f116ead455 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 20 08:51:00.861411 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:51:00.861411 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:51:00.861411 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:51:00.861411 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:51:00.861411 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:51:00.861411 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:51:00.861411 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:51:00.861411 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:51:00.861411 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:51:00.861411 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:51:00.861411 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:51:00.861411 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 20 08:51:00.861411 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 20 08:51:00.861411 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 20 08:51:00.861411 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 20 08:51:00.866696 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 20 08:51:00.866696 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 20 08:51:00.866696 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 20 08:51:00.866696 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 20 08:51:00.866696 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:51:00.866696 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:51:00.866696 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:51:00.866696 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:51:00.866696 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:51:00.866696 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:51:00.866696 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:51:00.866696 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:51:00.866696 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:51:00.866696 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:51:00.866696 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:51:00.866696 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:51:00.867630 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:51:00.867630 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:51:00.867630 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:51:00.867630 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:51:00.867630 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 20 08:51:00.867630 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 20 08:51:00.867630 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 20 08:51:00.867630 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 20 08:51:00.867630 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 20 08:51:00.867630 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 20 08:51:00.867630 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 20 08:51:00.867630 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:51:00.867630 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 463/913] 10.222.0.36 () {56 vars in 1102 bytes} [Thu Oct 20 08:51:00 2022] GET /identity/v3/roles/service => generated 85 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:51:00.871294 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc73fadd-43b5-4908-9a8e-8d25989ea1c8 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:51:00.917550 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:51:00.990800 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc73fadd-43b5-4908-9a8e-8d25989ea1c8 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:51:01.031896 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1da6b263-d1ca-4388-b65c-71743364d6d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:51:01.032511 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da6b263-d1ca-4388-b65c-71743364d6d5 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:01.035357 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da6b263-d1ca-4388-b65c-71743364d6d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:01.035357 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da6b263-d1ca-4388-b65c-71743364d6d5 None admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:01.035357 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1da6b263-d1ca-4388-b65c-71743364d6d5 None admin] RBAC: Adding query filter params (name=service) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:51:01.035357 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1da6b263-d1ca-4388-b65c-71743364d6d5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:51:01.035708 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1da6b263-d1ca-4388-b65c-71743364d6d5 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:51:01.044293 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 451/914] 10.222.0.36 () {56 vars in 1103 bytes} [Thu Oct 20 08:51:00 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:51:01.050159 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8138992f-6a75-47cd-9ef8-48f116ead455 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:51:01.108699 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:51:01.177343 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8138992f-6a75-47cd-9ef8-48f116ead455 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:51:01.217779 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d57f6a7-2e22-4b09-9646-42cf97e2b0a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:51:01.218779 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d57f6a7-2e22-4b09-9646-42cf97e2b0a2 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:01.219194 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d57f6a7-2e22-4b09-9646-42cf97e2b0a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:01.219646 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d57f6a7-2e22-4b09-9646-42cf97e2b0a2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:01.233165 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d57f6a7-2e22-4b09-9646-42cf97e2b0a2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:51:01.234858 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d57f6a7-2e22-4b09-9646-42cf97e2b0a2 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:51:01.251587 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-3d57f6a7-2e22-4b09-9646-42cf97e2b0a2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:51:01.251587 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:51:01.251587 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:51:01.251587 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:51:01.251587 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:51:01.251587 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:51:01.251587 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:51:01.251587 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:51:01.251587 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:51:01.251587 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:51:01.251587 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:51:01.251587 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:51:01.251587 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 20 08:51:01.251587 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 20 08:51:01.251587 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 20 08:51:01.251587 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 20 08:51:01.252809 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:51:01.252809 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:51:01.252809 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:51:01.252809 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:51:01.252809 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:51:01.252809 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:51:01.252809 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:51:01.252809 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:51:01.252809 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:51:01.252809 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:51:01.252809 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:51:01.252809 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:51:01.252809 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:51:01.252809 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:51:01.252809 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:51:01.252809 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:51:01.252809 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 20 08:51:01.254039 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 20 08:51:01.254039 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:51:01.254039 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:51:01.255533 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 464/915] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:51:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:51:01.262312 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1da6b263-d1ca-4388-b65c-71743364d6d5 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:51:01.389245 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:51:01.392859 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1da6b263-d1ca-4388-b65c-71743364d6d5 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:51:01.434148 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c8003c0-87da-4bb9-b5f1-61dff6257a60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:51:01.434909 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c8003c0-87da-4bb9-b5f1-61dff6257a60 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:01.435103 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c8003c0-87da-4bb9-b5f1-61dff6257a60 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:01.435375 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c8003c0-87da-4bb9-b5f1-61dff6257a60 None admin] PATH_INFO: `/v3/domains` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:01.436055 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c8003c0-87da-4bb9-b5f1-61dff6257a60 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:51:01.436373 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c8003c0-87da-4bb9-b5f1-61dff6257a60 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:51:01.437989 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c8003c0-87da-4bb9-b5f1-61dff6257a60 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:51:01.452037 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 452/916] 10.222.0.36 () {56 vars in 1109 bytes} [Thu Oct 20 08:51:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:51:01.458018 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d57f6a7-2e22-4b09-9646-42cf97e2b0a2 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:51:01.571899 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d57f6a7-2e22-4b09-9646-42cf97e2b0a2 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:51:01.573463 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:51:01.608820 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bf3c8f0-754b-4161-a387-b42cf7cff580 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:51:01.609394 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf3c8f0-754b-4161-a387-b42cf7cff580 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:01.609674 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf3c8f0-754b-4161-a387-b42cf7cff580 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:01.609965 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf3c8f0-754b-4161-a387-b42cf7cff580 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:01.617464 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bf3c8f0-754b-4161-a387-b42cf7cff580 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:51:01.618942 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bf3c8f0-754b-4161-a387-b42cf7cff580 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:51:01.630717 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-7bf3c8f0-754b-4161-a387-b42cf7cff580 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 20 08:51:01.630717 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:51:01.630717 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:51:01.630717 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:51:01.630717 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:51:01.630717 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:51:01.630717 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:51:01.630717 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:51:01.630717 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:51:01.630717 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:51:01.630717 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:51:01.630717 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:51:01.630717 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 20 08:51:01.630717 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 20 08:51:01.630717 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 20 08:51:01.630717 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 20 08:51:01.631784 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:51:01.631784 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:51:01.631784 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 20 08:51:01.631784 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 20 08:51:01.631784 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 20 08:51:01.631784 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 20 08:51:01.631784 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:51:01.631784 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:51:01.631784 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:51:01.631784 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:51:01.631784 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:51:01.631784 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:51:01.631784 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:51:01.631784 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:51:01.631784 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:51:01.631784 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:51:01.631784 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:51:01.631784 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:51:01.632761 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:51:01.632761 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:51:01.632761 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 20 08:51:01.632761 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 20 08:51:01.632761 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 20 08:51:01.632761 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 20 08:51:01.632761 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 20 08:51:01.632761 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 20 08:51:01.632761 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 20 08:51:01.632761 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:51:01.632761 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 465/917] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:51:01 2022] GET /identity/v3/users/placement => generated 87 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:51:01.637922 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c8003c0-87da-4bb9-b5f1-61dff6257a60 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:51:01.750660 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c8003c0-87da-4bb9-b5f1-61dff6257a60 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:51:01.788840 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-613f1fae-5bc4-4547-b0d4-4c6e89138209 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:51:01.789445 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-613f1fae-5bc4-4547-b0d4-4c6e89138209 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:01.789721 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-613f1fae-5bc4-4547-b0d4-4c6e89138209 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:01.789990 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-613f1fae-5bc4-4547-b0d4-4c6e89138209 None admin] PATH_INFO: `/v3/users` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:01.790741 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-613f1fae-5bc4-4547-b0d4-4c6e89138209 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:51:01.791042 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-613f1fae-5bc4-4547-b0d4-4c6e89138209 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:51:01.792502 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-613f1fae-5bc4-4547-b0d4-4c6e89138209 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:51:01.810121 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 453/918] 10.222.0.36 () {56 vars in 1143 bytes} [Thu Oct 20 08:51:01 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 08:51:01.815927 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bf3c8f0-754b-4161-a387-b42cf7cff580 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:51:01.930164 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bf3c8f0-754b-4161-a387-b42cf7cff580 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:51:01.968574 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a139a36-754b-42d6-b3d8-aba198dc1825 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:51:01.969242 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a139a36-754b-42d6-b3d8-aba198dc1825 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:01.969559 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a139a36-754b-42d6-b3d8-aba198dc1825 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:01.969836 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a139a36-754b-42d6-b3d8-aba198dc1825 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:01.982213 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a139a36-754b-42d6-b3d8-aba198dc1825 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:51:01.984022 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a139a36-754b-42d6-b3d8-aba198dc1825 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:51:02.000433 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-2a139a36-754b-42d6-b3d8-aba198dc1825 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:51:02.000433 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:51:02.000433 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:51:02.000433 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:51:02.000433 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:51:02.000433 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:51:02.000433 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:51:02.000433 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:51:02.000433 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:51:02.000433 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:51:02.000433 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:51:02.000433 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:51:02.000433 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 20 08:51:02.000433 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 20 08:51:02.000433 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 20 08:51:02.000433 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 20 08:51:02.001394 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:51:02.001394 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:51:02.001394 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:51:02.001394 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:51:02.001394 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:51:02.001394 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:51:02.001394 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:51:02.001394 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:51:02.001394 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:51:02.001394 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:51:02.001394 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:51:02.001394 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:51:02.001394 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:51:02.001394 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:51:02.001394 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:51:02.001394 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:51:02.001394 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 20 08:51:02.002283 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 20 08:51:02.002283 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:51:02.002283 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:51:02.007497 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 466/919] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:51:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:51:02.008565 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-613f1fae-5bc4-4547-b0d4-4c6e89138209 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:51:02.029163 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:51:02.134822 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-613f1fae-5bc4-4547-b0d4-4c6e89138209 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:51:02.173562 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9fe7f38-ceb0-4af2-ac4d-deafb0b5d70b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:51:02.174175 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9fe7f38-ceb0-4af2-ac4d-deafb0b5d70b None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:02.174450 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9fe7f38-ceb0-4af2-ac4d-deafb0b5d70b None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:02.174796 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9fe7f38-ceb0-4af2-ac4d-deafb0b5d70b None admin] PATH_INFO: `/v3/domains` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:02.175395 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9fe7f38-ceb0-4af2-ac4d-deafb0b5d70b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:51:02.175707 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9fe7f38-ceb0-4af2-ac4d-deafb0b5d70b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:51:02.178043 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9fe7f38-ceb0-4af2-ac4d-deafb0b5d70b None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:51:02.189011 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 454/920] 10.222.0.36 () {56 vars in 1109 bytes} [Thu Oct 20 08:51:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:51:02.195166 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a139a36-754b-42d6-b3d8-aba198dc1825 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:51:02.213531 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:51:02.308830 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a139a36-754b-42d6-b3d8-aba198dc1825 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:51:02.349449 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5214007-de9e-4f8a-8542-f8ef75a8bddc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:51:02.350131 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5214007-de9e-4f8a-8542-f8ef75a8bddc None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:02.350412 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5214007-de9e-4f8a-8542-f8ef75a8bddc None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:02.350746 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5214007-de9e-4f8a-8542-f8ef75a8bddc None admin] PATH_INFO: `/v3/projects/service` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:02.358049 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5214007-de9e-4f8a-8542-f8ef75a8bddc None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:51:02.359579 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5214007-de9e-4f8a-8542-f8ef75a8bddc None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:51:02.371545 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-d5214007-de9e-4f8a-8542-f8ef75a8bddc None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 20 08:51:02.371545 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:51:02.371545 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:51:02.371545 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:51:02.371545 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:51:02.371545 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:51:02.371545 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:51:02.371545 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:51:02.371545 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:51:02.371545 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:51:02.371545 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:51:02.371545 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:51:02.371545 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 20 08:51:02.371545 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 20 08:51:02.371545 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 20 08:51:02.372504 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 20 08:51:02.372504 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:51:02.372504 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:51:02.372504 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:51:02.372504 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:51:02.372504 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:51:02.372504 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:51:02.372504 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:51:02.372504 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:51:02.372504 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:51:02.372504 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:51:02.372504 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:51:02.372504 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:51:02.372504 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:51:02.372504 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:51:02.372504 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:51:02.372504 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:51:02.373418 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 20 08:51:02.373418 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 20 08:51:02.373418 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 20 08:51:02.373418 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 20 08:51:02.373418 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 20 08:51:02.373418 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 20 08:51:02.373418 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 20 08:51:02.373418 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:51:02.380240 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9fe7f38-ceb0-4af2-ac4d-deafb0b5d70b None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:51:02.380429 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 467/921] 10.222.0.36 () {56 vars in 1111 bytes} [Thu Oct 20 08:51:02 2022] GET /identity/v3/projects/service => generated 88 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:51:02.469021 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:51:02.498927 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9fe7f38-ceb0-4af2-ac4d-deafb0b5d70b None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:51:02.538601 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c6e0ef6-94ef-41a8-8a6b-767f7faee804 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:51:02.539155 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c6e0ef6-94ef-41a8-8a6b-767f7faee804 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:02.539432 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c6e0ef6-94ef-41a8-8a6b-767f7faee804 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:02.539730 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c6e0ef6-94ef-41a8-8a6b-767f7faee804 None admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:02.540400 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c6e0ef6-94ef-41a8-8a6b-767f7faee804 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:51:02.540692 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c6e0ef6-94ef-41a8-8a6b-767f7faee804 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:51:02.542132 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c6e0ef6-94ef-41a8-8a6b-767f7faee804 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:51:02.555719 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 455/922] 10.222.0.36 () {56 vars in 1148 bytes} [Thu Oct 20 08:51:02 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:51:02.562160 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5214007-de9e-4f8a-8542-f8ef75a8bddc None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:51:02.654945 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:51:02.682341 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5214007-de9e-4f8a-8542-f8ef75a8bddc None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:51:02.718878 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-895c3ba8-4df7-468c-b7e6-58a128b24475 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:51:02.719588 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895c3ba8-4df7-468c-b7e6-58a128b24475 None admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:02.719801 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895c3ba8-4df7-468c-b7e6-58a128b24475 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:02.720065 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895c3ba8-4df7-468c-b7e6-58a128b24475 None admin] PATH_INFO: `/v3/projects/d9857b610fb542969cfb5efbbdb8ce61/users/ad03ab76b9c44b1cb369475afa25eddd/roles/f81100d13381464aa48cbae3561ca1bc` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:02.736375 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-895c3ba8-4df7-468c-b7e6-58a128b24475 None admin] RBAC: Authorizing `identity:create_grant(project_id=d9857b610fb542969cfb5efbbdb8ce61, user_id=ad03ab76b9c44b1cb369475afa25eddd, role_id=f81100d13381464aa48cbae3561ca1bc)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:51:02.737804 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-895c3ba8-4df7-468c-b7e6-58a128b24475 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:51:02.751653 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 468/923] 10.222.0.36 () {58 vars in 1439 bytes} [Thu Oct 20 08:51:02 2022] PUT /identity/v3/projects/d9857b610fb542969cfb5efbbdb8ce61/users/ad03ab76b9c44b1cb369475afa25eddd/roles/f81100d13381464aa48cbae3561ca1bc => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 08:51:04.945732 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe91ab6d-9cbd-4b60-bae1-24169bb3fb00 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:04.946320 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe91ab6d-9cbd-4b60-bae1-24169bb3fb00 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:04.946320 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe91ab6d-9cbd-4b60-bae1-24169bb3fb00 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:04.947628 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 456/924] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:51:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:51:04.952841 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1c57313-c5a1-4ca1-ae77-c2721287f11f None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:04.953263 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1c57313-c5a1-4ca1-ae77-c2721287f11f None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:04.953388 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1c57313-c5a1-4ca1-ae77-c2721287f11f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:04.972346 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-b1c57313-c5a1-4ca1-ae77-c2721287f11f None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:51:04.973551 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-b1c57313-c5a1-4ca1-ae77-c2721287f11f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:51:05.041254 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 469/925] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:51:04 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 20 08:51:05.313382 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe91ab6d-9cbd-4b60-bae1-24169bb3fb00 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:51:05.432216 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe91ab6d-9cbd-4b60-bae1-24169bb3fb00 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:51:05.470618 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-301c0bc9-7f9a-4a92-a787-664bc96272ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:51:05.471249 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301c0bc9-7f9a-4a92-a787-664bc96272ed None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:05.471523 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301c0bc9-7f9a-4a92-a787-664bc96272ed None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:05.471793 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301c0bc9-7f9a-4a92-a787-664bc96272ed None admin] PATH_INFO: `/v3/roles/service` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:05.478578 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:51:05.482670 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-301c0bc9-7f9a-4a92-a787-664bc96272ed None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 20 08:51:05.490193 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-301c0bc9-7f9a-4a92-a787-664bc96272ed None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 20 08:51:05.490193 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:51:05.490193 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:51:05.490193 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:51:05.490193 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:51:05.490193 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:51:05.490193 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:51:05.490193 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:51:05.490193 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:51:05.490193 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:51:05.490193 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:51:05.490193 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:51:05.490193 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 20 08:51:05.490193 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 20 08:51:05.490193 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 20 08:51:05.490193 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 20 08:51:05.491325 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 20 08:51:05.491325 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 20 08:51:05.491325 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 20 08:51:05.491325 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 20 08:51:05.491325 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:51:05.491325 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:51:05.491325 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:51:05.491325 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:51:05.491325 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:51:05.491325 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:51:05.491325 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:51:05.491325 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:51:05.491325 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:51:05.491325 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:51:05.491325 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:51:05.491325 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:51:05.492544 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:51:05.492544 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:51:05.492544 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:51:05.492544 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:51:05.492544 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 20 08:51:05.492544 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 20 08:51:05.492544 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 20 08:51:05.492544 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 20 08:51:05.492544 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 20 08:51:05.492544 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 20 08:51:05.492544 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 20 08:51:05.492544 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:51:05.492544 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 457/926] 10.222.0.36 () {56 vars in 1102 bytes} [Thu Oct 20 08:51:05 2022] GET /identity/v3/roles/service => generated 85 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:51:05.497611 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1c57313-c5a1-4ca1-ae77-c2721287f11f None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:51:05.563222 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:51:05.611125 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1c57313-c5a1-4ca1-ae77-c2721287f11f None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:51:05.648406 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd6382b2-467a-4ffb-befe-bc58ababb666 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:51:05.649005 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6382b2-467a-4ffb-befe-bc58ababb666 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:05.649281 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6382b2-467a-4ffb-befe-bc58ababb666 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:05.649554 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6382b2-467a-4ffb-befe-bc58ababb666 None admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:05.650295 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd6382b2-467a-4ffb-befe-bc58ababb666 None admin] RBAC: Adding query filter params (name=service) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:51:05.650644 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd6382b2-467a-4ffb-befe-bc58ababb666 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:51:05.652161 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd6382b2-467a-4ffb-befe-bc58ababb666 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:51:05.660651 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 470/927] 10.222.0.36 () {56 vars in 1103 bytes} [Thu Oct 20 08:51:05 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 08:51:05.666910 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-301c0bc9-7f9a-4a92-a787-664bc96272ed None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:51:05.780648 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-301c0bc9-7f9a-4a92-a787-664bc96272ed None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:51:05.818659 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bd3d2e1-2d25-4206-b4e3-df66f2dbb603 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:51:05.819307 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd3d2e1-2d25-4206-b4e3-df66f2dbb603 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:05.819566 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd3d2e1-2d25-4206-b4e3-df66f2dbb603 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:05.819831 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd3d2e1-2d25-4206-b4e3-df66f2dbb603 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:05.831065 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bd3d2e1-2d25-4206-b4e3-df66f2dbb603 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:51:05.832892 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bd3d2e1-2d25-4206-b4e3-df66f2dbb603 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:51:05.849615 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-8bd3d2e1-2d25-4206-b4e3-df66f2dbb603 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:51:05.849615 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:51:05.849615 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:51:05.849615 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:51:05.849615 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:51:05.849615 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:51:05.849615 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:51:05.849615 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:51:05.849615 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:51:05.849615 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:51:05.849615 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:51:05.849615 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:51:05.849615 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 20 08:51:05.849615 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 20 08:51:05.849615 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 20 08:51:05.849615 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 20 08:51:05.850701 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:51:05.850701 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:51:05.850701 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:51:05.850701 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:51:05.850701 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:51:05.850701 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:51:05.850701 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:51:05.850701 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:51:05.850701 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:51:05.850701 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:51:05.850701 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:51:05.850701 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:51:05.850701 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:51:05.850701 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:51:05.850701 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:51:05.850701 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:51:05.850701 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 20 08:51:05.851952 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 20 08:51:05.851952 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:51:05.851952 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:51:05.851952 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 458/928] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:51:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 20 08:51:05.856727 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd6382b2-467a-4ffb-befe-bc58ababb666 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:51:05.963238 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd6382b2-467a-4ffb-befe-bc58ababb666 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:51:05.980212 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:51:05.998985 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff2936a-74ff-46fd-8edf-ae4066885dac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:51:05.999583 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff2936a-74ff-46fd-8edf-ae4066885dac None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:05.999857 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff2936a-74ff-46fd-8edf-ae4066885dac None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:06.000121 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff2936a-74ff-46fd-8edf-ae4066885dac None admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:06.000748 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bff2936a-74ff-46fd-8edf-ae4066885dac None admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:51:06.001036 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bff2936a-74ff-46fd-8edf-ae4066885dac None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:51:06.002390 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bff2936a-74ff-46fd-8edf-ae4066885dac None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:51:06.014388 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 471/929] 10.222.0.36 () {56 vars in 1109 bytes} [Thu Oct 20 08:51:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:51:06.020874 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bd3d2e1-2d25-4206-b4e3-df66f2dbb603 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:51:06.076366 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:51:06.135356 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bd3d2e1-2d25-4206-b4e3-df66f2dbb603 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:51:06.171102 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a218b368-c2c7-4173-9ec5-1601bfa243ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:51:06.171724 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a218b368-c2c7-4173-9ec5-1601bfa243ce None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:06.172019 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a218b368-c2c7-4173-9ec5-1601bfa243ce None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:06.172283 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a218b368-c2c7-4173-9ec5-1601bfa243ce None admin] PATH_INFO: `/v3/users/placement` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:06.180963 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a218b368-c2c7-4173-9ec5-1601bfa243ce None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:51:06.182727 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a218b368-c2c7-4173-9ec5-1601bfa243ce None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:51:06.194988 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-a218b368-c2c7-4173-9ec5-1601bfa243ce None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 20 08:51:06.194988 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:51:06.194988 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:51:06.194988 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:51:06.194988 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:51:06.194988 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:51:06.194988 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:51:06.194988 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:51:06.194988 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:51:06.194988 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:51:06.194988 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:51:06.194988 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:51:06.194988 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 20 08:51:06.194988 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 20 08:51:06.194988 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 20 08:51:06.194988 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 20 08:51:06.195984 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:51:06.195984 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:51:06.195984 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 20 08:51:06.195984 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 20 08:51:06.195984 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 20 08:51:06.195984 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 20 08:51:06.195984 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:51:06.195984 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:51:06.195984 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:51:06.195984 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:51:06.195984 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:51:06.195984 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:51:06.195984 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:51:06.195984 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:51:06.195984 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:51:06.195984 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:51:06.195984 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:51:06.195984 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:51:06.197017 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:51:06.197017 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:51:06.197017 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 20 08:51:06.197017 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 20 08:51:06.197017 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 20 08:51:06.197017 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 20 08:51:06.197017 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 20 08:51:06.197017 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 20 08:51:06.197017 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 20 08:51:06.197017 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:51:06.197017 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 459/930] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:51:06 2022] GET /identity/v3/users/placement => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:51:06.202345 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff2936a-74ff-46fd-8edf-ae4066885dac None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:51:06.314324 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff2936a-74ff-46fd-8edf-ae4066885dac None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:51:06.352083 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9dbcb99-c487-4d88-abcf-f3195f49dcdb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:51:06.352727 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9dbcb99-c487-4d88-abcf-f3195f49dcdb None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:06.353003 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9dbcb99-c487-4d88-abcf-f3195f49dcdb None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:06.353292 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9dbcb99-c487-4d88-abcf-f3195f49dcdb None admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:06.354068 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9dbcb99-c487-4d88-abcf-f3195f49dcdb None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:51:06.354370 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9dbcb99-c487-4d88-abcf-f3195f49dcdb None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:51:06.356059 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9dbcb99-c487-4d88-abcf-f3195f49dcdb None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:51:06.374258 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 472/931] 10.222.0.36 () {56 vars in 1143 bytes} [Thu Oct 20 08:51:06 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 08:51:06.380172 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a218b368-c2c7-4173-9ec5-1601bfa243ce None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:51:06.485606 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a218b368-c2c7-4173-9ec5-1601bfa243ce None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:51:06.496510 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:51:06.522972 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86d7573e-91bf-4653-a093-d8cfdfff7700 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:51:06.523824 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d7573e-91bf-4653-a093-d8cfdfff7700 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:06.524277 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d7573e-91bf-4653-a093-d8cfdfff7700 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:06.524533 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d7573e-91bf-4653-a093-d8cfdfff7700 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:06.595134 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86d7573e-91bf-4653-a093-d8cfdfff7700 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:51:06.596513 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86d7573e-91bf-4653-a093-d8cfdfff7700 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:51:06.699052 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 460/932] 10.222.0.36 () {58 vars in 1282 bytes} [Thu Oct 20 08:51:06 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 319 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 20 08:51:06.704891 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9dbcb99-c487-4d88-abcf-f3195f49dcdb None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:51:06.741882 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:51:06.810572 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9dbcb99-c487-4d88-abcf-f3195f49dcdb None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:51:06.845745 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55369c81-90be-44c9-abe4-0070b40e117a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:51:06.846302 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55369c81-90be-44c9-abe4-0070b40e117a None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:06.846646 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55369c81-90be-44c9-abe4-0070b40e117a None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:06.846886 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55369c81-90be-44c9-abe4-0070b40e117a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:06.857142 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55369c81-90be-44c9-abe4-0070b40e117a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:51:06.858614 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55369c81-90be-44c9-abe4-0070b40e117a None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:51:06.873781 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-55369c81-90be-44c9-abe4-0070b40e117a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:51:06.873781 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:51:06.873781 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:51:06.873781 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:51:06.873781 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:51:06.873781 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:51:06.873781 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:51:06.873781 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:51:06.873781 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:51:06.873781 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:51:06.873781 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:51:06.873781 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:51:06.873781 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 20 08:51:06.873781 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 20 08:51:06.873781 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 20 08:51:06.873781 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 20 08:51:06.874961 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:51:06.874961 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:51:06.874961 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:51:06.874961 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:51:06.874961 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:51:06.874961 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:51:06.874961 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:51:06.874961 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:51:06.874961 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:51:06.874961 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:51:06.874961 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:51:06.874961 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:51:06.874961 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:51:06.874961 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:51:06.874961 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:51:06.874961 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:51:06.874961 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 20 08:51:06.876062 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 20 08:51:06.876062 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:51:06.876062 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:51:06.876062 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 473/933] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:51:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 20 08:51:06.883325 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86d7573e-91bf-4653-a093-d8cfdfff7700 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:51:06.895128 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:51:06.993094 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86d7573e-91bf-4653-a093-d8cfdfff7700 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:51:07.031563 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7caf754e-2a79-414f-b916-e979a57b421f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:51:07.032217 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7caf754e-2a79-414f-b916-e979a57b421f None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:07.032435 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7caf754e-2a79-414f-b916-e979a57b421f None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:07.032707 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7caf754e-2a79-414f-b916-e979a57b421f None admin] PATH_INFO: `/v3/domains` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:07.033350 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7caf754e-2a79-414f-b916-e979a57b421f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:51:07.033649 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7caf754e-2a79-414f-b916-e979a57b421f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:51:07.035089 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7caf754e-2a79-414f-b916-e979a57b421f None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:51:07.046389 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 461/934] 10.222.0.36 () {56 vars in 1109 bytes} [Thu Oct 20 08:51:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:51:07.053317 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55369c81-90be-44c9-abe4-0070b40e117a None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:51:07.146973 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:51:07.162877 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55369c81-90be-44c9-abe4-0070b40e117a None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:51:07.198873 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f52e5088-0902-40a2-9710-765d32698268 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:51:07.199618 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52e5088-0902-40a2-9710-765d32698268 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:07.200028 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52e5088-0902-40a2-9710-765d32698268 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:07.200368 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52e5088-0902-40a2-9710-765d32698268 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:07.208374 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f52e5088-0902-40a2-9710-765d32698268 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:51:07.209824 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f52e5088-0902-40a2-9710-765d32698268 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:51:07.223399 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-f52e5088-0902-40a2-9710-765d32698268 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 20 08:51:07.223399 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:51:07.223399 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:51:07.223399 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:51:07.223399 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:51:07.223399 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:51:07.223399 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:51:07.223399 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:51:07.223399 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:51:07.223399 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:51:07.223399 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:51:07.223399 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:51:07.223399 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 20 08:51:07.223399 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 20 08:51:07.223399 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 20 08:51:07.224544 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 20 08:51:07.224544 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:51:07.224544 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:51:07.224544 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:51:07.224544 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:51:07.224544 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:51:07.224544 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:51:07.224544 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:51:07.224544 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:51:07.224544 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:51:07.224544 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:51:07.224544 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:51:07.224544 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:51:07.224544 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:51:07.224544 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:51:07.224544 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:51:07.224544 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:51:07.225647 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 20 08:51:07.225647 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 20 08:51:07.225647 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 20 08:51:07.225647 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 20 08:51:07.225647 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 20 08:51:07.225647 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 20 08:51:07.225647 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 20 08:51:07.225647 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:51:07.226120 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 474/935] 10.222.0.36 () {56 vars in 1111 bytes} [Thu Oct 20 08:51:07 2022] GET /identity/v3/projects/service => generated 88 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:51:07.231720 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7caf754e-2a79-414f-b916-e979a57b421f None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:51:07.306731 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:51:07.341501 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7caf754e-2a79-414f-b916-e979a57b421f None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:51:07.383427 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09ae1a48-f79c-4341-8b7b-682f350742e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:51:07.384110 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ae1a48-f79c-4341-8b7b-682f350742e7 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:07.384369 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ae1a48-f79c-4341-8b7b-682f350742e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:07.384681 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ae1a48-f79c-4341-8b7b-682f350742e7 None admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:07.385390 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09ae1a48-f79c-4341-8b7b-682f350742e7 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:51:07.385721 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09ae1a48-f79c-4341-8b7b-682f350742e7 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:51:07.387398 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09ae1a48-f79c-4341-8b7b-682f350742e7 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:51:07.400896 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 462/936] 10.222.0.36 () {56 vars in 1148 bytes} [Thu Oct 20 08:51:07 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:51:07.407603 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f52e5088-0902-40a2-9710-765d32698268 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:51:07.513144 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f52e5088-0902-40a2-9710-765d32698268 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:51:07.548286 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1944a799-45b7-4d50-8ab1-fcda53a8fad3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:51:07.548882 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1944a799-45b7-4d50-8ab1-fcda53a8fad3 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:07.549160 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1944a799-45b7-4d50-8ab1-fcda53a8fad3 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:07.549431 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1944a799-45b7-4d50-8ab1-fcda53a8fad3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:07.550182 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1944a799-45b7-4d50-8ab1-fcda53a8fad3 None admin] RBAC: Adding query filter params (role.id=f81100d13381464aa48cbae3561ca1bc, scope.project.id=d9857b610fb542969cfb5efbbdb8ce61, user.id=ad03ab76b9c44b1cb369475afa25eddd) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:51:07.550611 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1944a799-45b7-4d50-8ab1-fcda53a8fad3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:51:07.552506 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1944a799-45b7-4d50-8ab1-fcda53a8fad3 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:51:07.559162 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 475/937] 10.222.0.36 () {56 vars in 1374 bytes} [Thu Oct 20 08:51:07 2022] GET /identity/v3/role_assignments?user.id=ad03ab76b9c44b1cb369475afa25eddd&scope.project.id=d9857b610fb542969cfb5efbbdb8ce61&role.id=f81100d13381464aa48cbae3561ca1bc => generated 608 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:51:09.774245 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a21d814-34ca-42b3-bf6e-9db5b2439cb7 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:09.774837 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a21d814-34ca-42b3-bf6e-9db5b2439cb7 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:09.774837 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a21d814-34ca-42b3-bf6e-9db5b2439cb7 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:09.775982 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 463/938] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:51:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:51:09.781474 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fce7e25-3199-4fe3-8c56-d7c0fb88716a None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:09.781933 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fce7e25-3199-4fe3-8c56-d7c0fb88716a None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:09.782026 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fce7e25-3199-4fe3-8c56-d7c0fb88716a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:09.801784 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-8fce7e25-3199-4fe3-8c56-d7c0fb88716a None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:51:09.803009 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-8fce7e25-3199-4fe3-8c56-d7c0fb88716a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:51:09.805868 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:51:09.875527 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 476/939] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:51:09 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 20 08:51:10.164389 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a21d814-34ca-42b3-bf6e-9db5b2439cb7 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:51:10.278960 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a21d814-34ca-42b3-bf6e-9db5b2439cb7 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:51:10.307390 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:51:10.313619 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-382001bc-62a1-48da-b47a-c98a55443d8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:51:10.314238 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-382001bc-62a1-48da-b47a-c98a55443d8d None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:10.314539 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-382001bc-62a1-48da-b47a-c98a55443d8d None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:10.315750 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-382001bc-62a1-48da-b47a-c98a55443d8d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:10.326456 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-382001bc-62a1-48da-b47a-c98a55443d8d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 20 08:51:10.331171 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-382001bc-62a1-48da-b47a-c98a55443d8d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 20 08:51:10.331171 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:51:10.331171 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:51:10.331171 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:51:10.331171 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:51:10.331171 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:51:10.331171 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:51:10.331171 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:51:10.331171 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:51:10.331171 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:51:10.331171 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:51:10.331171 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:51:10.331171 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 20 08:51:10.331171 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 20 08:51:10.331171 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 20 08:51:10.331171 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 20 08:51:10.332853 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 20 08:51:10.332853 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 20 08:51:10.332853 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 20 08:51:10.332853 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 20 08:51:10.332853 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:51:10.332853 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:51:10.332853 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:51:10.332853 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:51:10.332853 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:51:10.332853 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:51:10.332853 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:51:10.332853 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:51:10.332853 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:51:10.332853 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:51:10.332853 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:51:10.332853 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:51:10.333701 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:51:10.333701 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:51:10.333701 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:51:10.333701 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:51:10.333701 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 20 08:51:10.333701 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 20 08:51:10.333701 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 20 08:51:10.333701 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 20 08:51:10.333701 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 20 08:51:10.333701 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 20 08:51:10.333701 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 20 08:51:10.333701 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:51:10.333701 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 464/940] 10.222.0.36 () {56 vars in 1096 bytes} [Thu Oct 20 08:51:10 2022] GET /identity/v3/roles/admin => generated 83 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:51:10.339601 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fce7e25-3199-4fe3-8c56-d7c0fb88716a None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:51:10.447264 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fce7e25-3199-4fe3-8c56-d7c0fb88716a None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:51:10.483494 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-327682cf-c1b6-4293-b14c-c3204cbc5e22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:51:10.484115 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327682cf-c1b6-4293-b14c-c3204cbc5e22 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:10.484371 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327682cf-c1b6-4293-b14c-c3204cbc5e22 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:10.484635 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327682cf-c1b6-4293-b14c-c3204cbc5e22 None admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:10.485287 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-327682cf-c1b6-4293-b14c-c3204cbc5e22 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:51:10.485583 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-327682cf-c1b6-4293-b14c-c3204cbc5e22 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:51:10.486979 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-327682cf-c1b6-4293-b14c-c3204cbc5e22 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:51:10.495030 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 477/941] 10.222.0.36 () {56 vars in 1099 bytes} [Thu Oct 20 08:51:10 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:51:10.500741 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-382001bc-62a1-48da-b47a-c98a55443d8d None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:51:10.604787 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-382001bc-62a1-48da-b47a-c98a55443d8d None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:51:10.643101 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7c8274b-a5cd-420a-a94a-ddf060479c3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:51:10.643747 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c8274b-a5cd-420a-a94a-ddf060479c3a None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:10.644077 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c8274b-a5cd-420a-a94a-ddf060479c3a None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:10.644323 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c8274b-a5cd-420a-a94a-ddf060479c3a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:10.654569 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7c8274b-a5cd-420a-a94a-ddf060479c3a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:51:10.656044 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7c8274b-a5cd-420a-a94a-ddf060479c3a None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:51:10.670623 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-c7c8274b-a5cd-420a-a94a-ddf060479c3a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:51:10.670623 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:51:10.670623 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:51:10.670623 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:51:10.670623 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:51:10.670623 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:51:10.670623 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:51:10.670623 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:51:10.670623 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:51:10.670623 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:51:10.670623 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:51:10.670623 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:51:10.670623 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 20 08:51:10.670623 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 20 08:51:10.670623 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 20 08:51:10.670623 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 20 08:51:10.671766 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:51:10.671766 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:51:10.671766 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:51:10.671766 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:51:10.671766 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:51:10.671766 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:51:10.671766 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:51:10.671766 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:51:10.671766 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:51:10.671766 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:51:10.671766 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:51:10.671766 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:51:10.671766 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:51:10.671766 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:51:10.671766 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:51:10.671766 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:51:10.671766 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 20 08:51:10.672808 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 20 08:51:10.672808 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:51:10.672808 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:51:10.672808 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 465/942] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:51:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:51:10.678216 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-327682cf-c1b6-4293-b14c-c3204cbc5e22 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:51:10.681353 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:51:10.789372 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-327682cf-c1b6-4293-b14c-c3204cbc5e22 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:51:10.826757 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffa118d2-a2c6-4929-9fb9-cf26a0f923be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:51:10.827442 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa118d2-a2c6-4929-9fb9-cf26a0f923be None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:10.828618 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa118d2-a2c6-4929-9fb9-cf26a0f923be None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:10.828618 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa118d2-a2c6-4929-9fb9-cf26a0f923be None admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:10.828810 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffa118d2-a2c6-4929-9fb9-cf26a0f923be None admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:51:10.829018 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffa118d2-a2c6-4929-9fb9-cf26a0f923be None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:51:10.830652 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffa118d2-a2c6-4929-9fb9-cf26a0f923be None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:51:10.844085 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 478/943] 10.222.0.36 () {56 vars in 1109 bytes} [Thu Oct 20 08:51:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:51:10.850700 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7c8274b-a5cd-420a-a94a-ddf060479c3a None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:51:10.896384 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:51:10.959568 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7c8274b-a5cd-420a-a94a-ddf060479c3a None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:51:10.994440 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff7d2eb-11e3-4c1e-8595-7230f8e285c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:51:10.995111 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff7d2eb-11e3-4c1e-8595-7230f8e285c1 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:10.995375 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff7d2eb-11e3-4c1e-8595-7230f8e285c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:10.995670 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff7d2eb-11e3-4c1e-8595-7230f8e285c1 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:11.004897 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fff7d2eb-11e3-4c1e-8595-7230f8e285c1 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:51:11.006576 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fff7d2eb-11e3-4c1e-8595-7230f8e285c1 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:51:11.018714 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-fff7d2eb-11e3-4c1e-8595-7230f8e285c1 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 20 08:51:11.018714 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:51:11.018714 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:51:11.018714 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:51:11.018714 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:51:11.018714 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:51:11.018714 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:51:11.018714 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:51:11.018714 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:51:11.018714 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:51:11.018714 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:51:11.018714 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:51:11.018714 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 20 08:51:11.018714 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 20 08:51:11.018714 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 20 08:51:11.018714 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 20 08:51:11.019760 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:51:11.019760 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:51:11.019760 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 20 08:51:11.019760 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 20 08:51:11.019760 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 20 08:51:11.019760 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 20 08:51:11.019760 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:51:11.019760 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:51:11.019760 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:51:11.019760 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:51:11.019760 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:51:11.019760 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:51:11.019760 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:51:11.019760 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:51:11.019760 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:51:11.019760 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:51:11.019760 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:51:11.019760 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:51:11.021171 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:51:11.021171 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:51:11.021171 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 20 08:51:11.021171 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 20 08:51:11.021171 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 20 08:51:11.021171 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 20 08:51:11.021171 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 20 08:51:11.021171 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 20 08:51:11.021171 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 20 08:51:11.021171 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:51:11.021171 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 466/944] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:51:10 2022] GET /identity/v3/users/placement => generated 87 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:51:11.026043 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffa118d2-a2c6-4929-9fb9-cf26a0f923be None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:51:11.093329 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:51:11.142270 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffa118d2-a2c6-4929-9fb9-cf26a0f923be None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:51:11.179716 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-938c90fa-939b-4f22-9220-2d64ea62ed2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:51:11.180313 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-938c90fa-939b-4f22-9220-2d64ea62ed2e None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:11.180585 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-938c90fa-939b-4f22-9220-2d64ea62ed2e None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:11.180849 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-938c90fa-939b-4f22-9220-2d64ea62ed2e None admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:11.181556 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-938c90fa-939b-4f22-9220-2d64ea62ed2e None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:51:11.181854 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-938c90fa-939b-4f22-9220-2d64ea62ed2e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:51:11.183289 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-938c90fa-939b-4f22-9220-2d64ea62ed2e None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:51:11.202800 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 479/945] 10.222.0.36 () {56 vars in 1143 bytes} [Thu Oct 20 08:51:11 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:51:11.208869 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff7d2eb-11e3-4c1e-8595-7230f8e285c1 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:51:11.314321 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff7d2eb-11e3-4c1e-8595-7230f8e285c1 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:51:11.319941 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:51:11.358958 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-988b9738-e326-4d2f-8e12-a585de7f3a74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:51:11.359600 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988b9738-e326-4d2f-8e12-a585de7f3a74 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:11.359901 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988b9738-e326-4d2f-8e12-a585de7f3a74 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:11.360231 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988b9738-e326-4d2f-8e12-a585de7f3a74 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:11.374346 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-988b9738-e326-4d2f-8e12-a585de7f3a74 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:51:11.375881 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-988b9738-e326-4d2f-8e12-a585de7f3a74 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:51:11.391184 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-988b9738-e326-4d2f-8e12-a585de7f3a74 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:51:11.391184 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:51:11.391184 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:51:11.391184 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:51:11.391184 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:51:11.391184 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:51:11.391184 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:51:11.391184 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:51:11.391184 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:51:11.391184 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:51:11.391184 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:51:11.391184 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:51:11.391184 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 20 08:51:11.391184 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 20 08:51:11.391184 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 20 08:51:11.391184 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 20 08:51:11.392403 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:51:11.392403 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:51:11.392403 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:51:11.392403 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:51:11.392403 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:51:11.392403 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:51:11.392403 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:51:11.392403 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:51:11.392403 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:51:11.392403 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:51:11.392403 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:51:11.392403 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:51:11.392403 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:51:11.392403 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:51:11.392403 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:51:11.392403 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:51:11.392403 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 20 08:51:11.393312 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 20 08:51:11.393312 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:51:11.393312 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:51:11.393312 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 467/946] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:51:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:51:11.398583 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-938c90fa-939b-4f22-9220-2d64ea62ed2e None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:51:11.507296 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-938c90fa-939b-4f22-9220-2d64ea62ed2e None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:51:11.528617 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:51:11.544366 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c157cce8-b086-41c3-a6e6-cb526e9d85b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:51:11.545032 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c157cce8-b086-41c3-a6e6-cb526e9d85b3 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:11.545343 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c157cce8-b086-41c3-a6e6-cb526e9d85b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:11.545667 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c157cce8-b086-41c3-a6e6-cb526e9d85b3 None admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:11.546368 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c157cce8-b086-41c3-a6e6-cb526e9d85b3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:51:11.546744 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c157cce8-b086-41c3-a6e6-cb526e9d85b3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:51:11.548248 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c157cce8-b086-41c3-a6e6-cb526e9d85b3 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:51:11.560057 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 480/947] 10.222.0.36 () {56 vars in 1109 bytes} [Thu Oct 20 08:51:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:51:11.565922 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-988b9738-e326-4d2f-8e12-a585de7f3a74 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:51:11.677144 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-988b9738-e326-4d2f-8e12-a585de7f3a74 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:51:11.714070 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8898bc7-9eee-4ced-9148-935155701812 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:51:11.714940 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8898bc7-9eee-4ced-9148-935155701812 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:11.715227 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8898bc7-9eee-4ced-9148-935155701812 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:11.715499 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8898bc7-9eee-4ced-9148-935155701812 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:11.723039 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8898bc7-9eee-4ced-9148-935155701812 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:51:11.724645 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8898bc7-9eee-4ced-9148-935155701812 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:51:11.736412 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-c8898bc7-9eee-4ced-9148-935155701812 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 20 08:51:11.736412 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:51:11.736412 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:51:11.736412 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:51:11.736412 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:51:11.736412 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:51:11.736412 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:51:11.736412 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:51:11.736412 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:51:11.736412 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:51:11.736412 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:51:11.736412 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:51:11.736412 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 20 08:51:11.736412 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 20 08:51:11.736412 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 20 08:51:11.737360 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 20 08:51:11.737360 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:51:11.737360 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:51:11.737360 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:51:11.737360 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:51:11.737360 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:51:11.737360 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:51:11.737360 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:51:11.737360 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:51:11.737360 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:51:11.737360 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:51:11.737360 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:51:11.737360 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:51:11.737360 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:51:11.737360 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:51:11.737360 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:51:11.737360 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:51:11.743757 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 20 08:51:11.743757 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 20 08:51:11.743757 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 20 08:51:11.743757 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 20 08:51:11.743757 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 20 08:51:11.743757 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 20 08:51:11.743757 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 20 08:51:11.743757 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:51:11.743757 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 468/948] 10.222.0.36 () {56 vars in 1111 bytes} [Thu Oct 20 08:51:11 2022] GET /identity/v3/projects/service => generated 88 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 20 08:51:11.744248 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c157cce8-b086-41c3-a6e6-cb526e9d85b3 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:51:11.858586 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c157cce8-b086-41c3-a6e6-cb526e9d85b3 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:51:11.894996 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b44816c-cda0-4429-bf6e-83dc0faf7a0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:51:11.895599 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b44816c-cda0-4429-bf6e-83dc0faf7a0b None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:11.895873 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b44816c-cda0-4429-bf6e-83dc0faf7a0b None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:11.896138 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b44816c-cda0-4429-bf6e-83dc0faf7a0b None admin] PATH_INFO: `/v3/projects` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:11.896816 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b44816c-cda0-4429-bf6e-83dc0faf7a0b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:51:11.897146 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b44816c-cda0-4429-bf6e-83dc0faf7a0b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:51:11.899065 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b44816c-cda0-4429-bf6e-83dc0faf7a0b None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:51:11.914620 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 481/949] 10.222.0.36 () {56 vars in 1148 bytes} [Thu Oct 20 08:51:11 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 08:51:11.920543 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8898bc7-9eee-4ced-9148-935155701812 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:51:11.939938 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:51:12.035601 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8898bc7-9eee-4ced-9148-935155701812 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:51:12.073084 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39eb8c06-0c09-46fa-a10b-7867c0033689 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:51:12.074022 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39eb8c06-0c09-46fa-a10b-7867c0033689 None admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:12.074349 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39eb8c06-0c09-46fa-a10b-7867c0033689 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:12.074657 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39eb8c06-0c09-46fa-a10b-7867c0033689 None admin] PATH_INFO: `/v3/projects/d9857b610fb542969cfb5efbbdb8ce61/users/ad03ab76b9c44b1cb369475afa25eddd/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:12.076772 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39eb8c06-0c09-46fa-a10b-7867c0033689 None admin] RBAC: Authorizing `identity:create_grant(project_id=d9857b610fb542969cfb5efbbdb8ce61, user_id=ad03ab76b9c44b1cb369475afa25eddd, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:51:12.078203 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39eb8c06-0c09-46fa-a10b-7867c0033689 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:51:12.095671 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 469/950] 10.222.0.36 () {58 vars in 1439 bytes} [Thu Oct 20 08:51:11 2022] PUT /identity/v3/projects/d9857b610fb542969cfb5efbbdb8ce61/users/ad03ab76b9c44b1cb369475afa25eddd/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 08:51:14.467607 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5641da3-9612-41d5-ab5c-936b80a76e58 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:14.468178 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5641da3-9612-41d5-ab5c-936b80a76e58 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:14.468178 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5641da3-9612-41d5-ab5c-936b80a76e58 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:14.469696 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 482/951] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:51:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:51:14.475746 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09013ee4-0203-4ac2-8b2d-d17eb71853f6 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:14.476205 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09013ee4-0203-4ac2-8b2d-d17eb71853f6 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:14.476482 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09013ee4-0203-4ac2-8b2d-d17eb71853f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:14.495593 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-09013ee4-0203-4ac2-8b2d-d17eb71853f6 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:51:14.496800 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-09013ee4-0203-4ac2-8b2d-d17eb71853f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:51:14.578559 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 470/952] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:51:14 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 20 08:51:14.840451 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5641da3-9612-41d5-ab5c-936b80a76e58 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:51:14.884030 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:51:14.950949 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5641da3-9612-41d5-ab5c-936b80a76e58 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:51:14.988695 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e1fd74-617c-4f78-9dc4-34c9d02122f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:51:14.989722 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e1fd74-617c-4f78-9dc4-34c9d02122f6 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:14.990080 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e1fd74-617c-4f78-9dc4-34c9d02122f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:14.990355 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e1fd74-617c-4f78-9dc4-34c9d02122f6 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:15.000680 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71e1fd74-617c-4f78-9dc4-34c9d02122f6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 20 08:51:15.005928 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-71e1fd74-617c-4f78-9dc4-34c9d02122f6 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 20 08:51:15.005928 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:51:15.005928 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:51:15.005928 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:51:15.005928 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:51:15.005928 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:51:15.005928 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:51:15.005928 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:51:15.005928 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:51:15.005928 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:51:15.005928 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:51:15.005928 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:51:15.005928 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 20 08:51:15.005928 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 20 08:51:15.005928 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 20 08:51:15.005928 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 20 08:51:15.007173 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 20 08:51:15.007173 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 20 08:51:15.007173 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 20 08:51:15.007173 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 20 08:51:15.007173 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:51:15.007173 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:51:15.007173 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:51:15.007173 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:51:15.007173 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:51:15.007173 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:51:15.007173 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:51:15.007173 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:51:15.007173 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:51:15.007173 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:51:15.007173 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:51:15.007173 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:51:15.008064 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:51:15.008064 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:51:15.008064 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:51:15.008064 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:51:15.008064 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 20 08:51:15.008064 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 20 08:51:15.008064 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 20 08:51:15.008064 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 20 08:51:15.008064 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 20 08:51:15.008064 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 20 08:51:15.008064 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 20 08:51:15.008064 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:51:15.008064 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 483/953] 10.222.0.36 () {56 vars in 1096 bytes} [Thu Oct 20 08:51:14 2022] GET /identity/v3/roles/admin => generated 83 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 20 08:51:15.013528 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09013ee4-0203-4ac2-8b2d-d17eb71853f6 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:51:15.016898 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:51:15.127662 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09013ee4-0203-4ac2-8b2d-d17eb71853f6 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:51:15.171179 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cff14f23-80f9-43b4-8bd1-dacb2cd6cb46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:51:15.171800 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff14f23-80f9-43b4-8bd1-dacb2cd6cb46 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:15.172067 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff14f23-80f9-43b4-8bd1-dacb2cd6cb46 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:15.172364 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff14f23-80f9-43b4-8bd1-dacb2cd6cb46 None admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:15.173024 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cff14f23-80f9-43b4-8bd1-dacb2cd6cb46 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:51:15.173305 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cff14f23-80f9-43b4-8bd1-dacb2cd6cb46 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:51:15.174678 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cff14f23-80f9-43b4-8bd1-dacb2cd6cb46 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:51:15.183175 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 471/954] 10.222.0.36 () {56 vars in 1099 bytes} [Thu Oct 20 08:51:15 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:51:15.189249 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e1fd74-617c-4f78-9dc4-34c9d02122f6 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:51:15.300085 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:51:15.304309 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e1fd74-617c-4f78-9dc4-34c9d02122f6 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:51:15.345088 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e81cc351-f9a7-47fc-8571-fa95c2662c06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:51:15.345719 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e81cc351-f9a7-47fc-8571-fa95c2662c06 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:15.346093 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e81cc351-f9a7-47fc-8571-fa95c2662c06 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:15.346516 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e81cc351-f9a7-47fc-8571-fa95c2662c06 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:15.357675 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e81cc351-f9a7-47fc-8571-fa95c2662c06 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:51:15.359154 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e81cc351-f9a7-47fc-8571-fa95c2662c06 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:51:15.376151 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-e81cc351-f9a7-47fc-8571-fa95c2662c06 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:51:15.376151 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:51:15.376151 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:51:15.376151 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:51:15.376151 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:51:15.376151 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:51:15.376151 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:51:15.376151 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:51:15.376151 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:51:15.376151 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:51:15.376151 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:51:15.376151 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:51:15.376151 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 20 08:51:15.376151 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 20 08:51:15.376151 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 20 08:51:15.376151 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 20 08:51:15.377331 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:51:15.377331 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:51:15.377331 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:51:15.377331 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:51:15.377331 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:51:15.377331 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:51:15.377331 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:51:15.377331 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:51:15.377331 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:51:15.377331 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:51:15.377331 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:51:15.377331 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:51:15.377331 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:51:15.377331 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:51:15.377331 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:51:15.377331 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:51:15.377331 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 20 08:51:15.378303 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 20 08:51:15.378303 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:51:15.378303 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:51:15.378303 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 484/955] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:51:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 20 08:51:15.386051 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cff14f23-80f9-43b4-8bd1-dacb2cd6cb46 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:51:15.460404 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:51:15.502552 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cff14f23-80f9-43b4-8bd1-dacb2cd6cb46 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:51:15.539273 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b9be1e3-15bc-44b5-b67c-e353b3bb3360 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:51:15.539844 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b9be1e3-15bc-44b5-b67c-e353b3bb3360 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:15.540226 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b9be1e3-15bc-44b5-b67c-e353b3bb3360 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:15.540469 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b9be1e3-15bc-44b5-b67c-e353b3bb3360 None admin] PATH_INFO: `/v3/domains` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:15.541113 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b9be1e3-15bc-44b5-b67c-e353b3bb3360 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:51:15.541412 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b9be1e3-15bc-44b5-b67c-e353b3bb3360 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:51:15.542821 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b9be1e3-15bc-44b5-b67c-e353b3bb3360 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:51:15.556174 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 472/956] 10.222.0.36 () {56 vars in 1109 bytes} [Thu Oct 20 08:51:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:51:15.562704 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e81cc351-f9a7-47fc-8571-fa95c2662c06 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:51:15.670625 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e81cc351-f9a7-47fc-8571-fa95c2662c06 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:51:15.705757 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49c0f396-6ea1-40e7-b2d5-fd26c2401f8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:51:15.706397 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c0f396-6ea1-40e7-b2d5-fd26c2401f8c None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:15.706729 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c0f396-6ea1-40e7-b2d5-fd26c2401f8c None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:15.707000 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c0f396-6ea1-40e7-b2d5-fd26c2401f8c None admin] PATH_INFO: `/v3/users/placement` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:15.715666 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49c0f396-6ea1-40e7-b2d5-fd26c2401f8c None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:51:15.717130 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49c0f396-6ea1-40e7-b2d5-fd26c2401f8c None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:51:15.729769 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-49c0f396-6ea1-40e7-b2d5-fd26c2401f8c None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 20 08:51:15.729769 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:51:15.729769 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:51:15.729769 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:51:15.729769 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:51:15.729769 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:51:15.729769 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:51:15.729769 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:51:15.729769 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:51:15.729769 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:51:15.729769 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:51:15.729769 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:51:15.729769 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 20 08:51:15.729769 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 20 08:51:15.729769 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 20 08:51:15.729769 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 20 08:51:15.730965 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:51:15.730965 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:51:15.730965 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 20 08:51:15.730965 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 20 08:51:15.730965 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 20 08:51:15.730965 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 20 08:51:15.730965 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:51:15.730965 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:51:15.730965 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:51:15.730965 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:51:15.730965 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:51:15.730965 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:51:15.730965 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:51:15.730965 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:51:15.730965 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:51:15.730965 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:51:15.730965 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:51:15.730965 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:51:15.731943 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:51:15.731943 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:51:15.731943 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 20 08:51:15.731943 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 20 08:51:15.731943 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 20 08:51:15.731943 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 20 08:51:15.731943 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 20 08:51:15.731943 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 20 08:51:15.731943 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 20 08:51:15.731943 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:51:15.731943 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 485/957] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:51:15 2022] GET /identity/v3/users/placement => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 20 08:51:15.736856 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b9be1e3-15bc-44b5-b67c-e353b3bb3360 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:51:15.841865 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b9be1e3-15bc-44b5-b67c-e353b3bb3360 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:51:15.869158 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:51:15.878551 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26c8b735-b7a0-481c-a9ba-ec83c727397e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:51:15.879170 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c8b735-b7a0-481c-a9ba-ec83c727397e None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:15.879445 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c8b735-b7a0-481c-a9ba-ec83c727397e None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:15.879712 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c8b735-b7a0-481c-a9ba-ec83c727397e None admin] PATH_INFO: `/v3/users` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:15.880444 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26c8b735-b7a0-481c-a9ba-ec83c727397e None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:51:15.880747 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26c8b735-b7a0-481c-a9ba-ec83c727397e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:51:15.882235 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26c8b735-b7a0-481c-a9ba-ec83c727397e None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:51:15.906642 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 473/958] 10.222.0.36 () {56 vars in 1143 bytes} [Thu Oct 20 08:51:15 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:51:15.907926 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49c0f396-6ea1-40e7-b2d5-fd26c2401f8c None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:51:15.924881 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:51:16.012807 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49c0f396-6ea1-40e7-b2d5-fd26c2401f8c None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:51:16.048067 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82adac79-a6e4-4a48-a80a-ee955d669825 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:51:16.048735 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82adac79-a6e4-4a48-a80a-ee955d669825 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:16.048943 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82adac79-a6e4-4a48-a80a-ee955d669825 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:16.049209 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82adac79-a6e4-4a48-a80a-ee955d669825 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:16.123440 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82adac79-a6e4-4a48-a80a-ee955d669825 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:51:16.124852 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82adac79-a6e4-4a48-a80a-ee955d669825 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:51:16.142567 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:51:16.231699 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 486/959] 10.222.0.36 () {58 vars in 1282 bytes} [Thu Oct 20 08:51:15 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 324 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 20 08:51:16.237286 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26c8b735-b7a0-481c-a9ba-ec83c727397e None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:51:16.340526 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26c8b735-b7a0-481c-a9ba-ec83c727397e None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:51:16.376377 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d8edf80-27e7-46e1-a46f-9ee99ea893f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:51:16.376993 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d8edf80-27e7-46e1-a46f-9ee99ea893f9 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:16.377268 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d8edf80-27e7-46e1-a46f-9ee99ea893f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:16.377536 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d8edf80-27e7-46e1-a46f-9ee99ea893f9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:16.387911 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d8edf80-27e7-46e1-a46f-9ee99ea893f9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:51:16.389349 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d8edf80-27e7-46e1-a46f-9ee99ea893f9 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:51:16.403917 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-9d8edf80-27e7-46e1-a46f-9ee99ea893f9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:51:16.403917 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:51:16.403917 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:51:16.403917 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:51:16.403917 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:51:16.403917 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:51:16.403917 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:51:16.403917 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:51:16.403917 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:51:16.403917 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:51:16.403917 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:51:16.403917 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:51:16.403917 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 20 08:51:16.403917 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 20 08:51:16.403917 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 20 08:51:16.403917 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 20 08:51:16.405056 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:51:16.405056 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:51:16.405056 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:51:16.405056 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:51:16.405056 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:51:16.405056 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:51:16.405056 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:51:16.405056 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:51:16.405056 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:51:16.405056 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:51:16.405056 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:51:16.405056 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:51:16.405056 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:51:16.405056 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:51:16.405056 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:51:16.405056 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:51:16.405056 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 20 08:51:16.406259 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 20 08:51:16.406259 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 20 08:51:16.406259 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:51:16.406259 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 474/960] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:51:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:51:16.410700 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82adac79-a6e4-4a48-a80a-ee955d669825 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:51:16.503972 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82adac79-a6e4-4a48-a80a-ee955d669825 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:51:16.516181 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:51:16.536846 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba25422e-2a77-44a6-a5ff-cea62607135a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:51:16.537426 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba25422e-2a77-44a6-a5ff-cea62607135a None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:16.537699 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba25422e-2a77-44a6-a5ff-cea62607135a None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:16.537959 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba25422e-2a77-44a6-a5ff-cea62607135a None admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:16.538602 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba25422e-2a77-44a6-a5ff-cea62607135a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:51:16.538897 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba25422e-2a77-44a6-a5ff-cea62607135a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:51:16.540233 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba25422e-2a77-44a6-a5ff-cea62607135a None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:51:16.550942 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 487/961] 10.222.0.36 () {56 vars in 1109 bytes} [Thu Oct 20 08:51:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:51:16.556457 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d8edf80-27e7-46e1-a46f-9ee99ea893f9 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:51:16.591537 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:51:16.653748 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d8edf80-27e7-46e1-a46f-9ee99ea893f9 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:51:16.686988 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-770ea5aa-7567-4c12-b14e-a31cd801b6fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:51:16.687589 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770ea5aa-7567-4c12-b14e-a31cd801b6fa None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:16.687861 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770ea5aa-7567-4c12-b14e-a31cd801b6fa None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:16.688134 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770ea5aa-7567-4c12-b14e-a31cd801b6fa None admin] PATH_INFO: `/v3/projects/service` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:16.695075 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-770ea5aa-7567-4c12-b14e-a31cd801b6fa None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:51:16.696498 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-770ea5aa-7567-4c12-b14e-a31cd801b6fa None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:51:16.707248 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application [None req-770ea5aa-7567-4c12-b14e-a31cd801b6fa None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 20 08:51:16.707248 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:51:16.707248 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:51:16.707248 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:51:16.707248 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:51:16.707248 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:51:16.707248 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:51:16.707248 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:51:16.707248 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:51:16.707248 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:51:16.707248 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:51:16.707248 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:51:16.707248 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 20 08:51:16.707248 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 20 08:51:16.707248 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 20 08:51:16.708244 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 20 08:51:16.708244 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:51:16.708244 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:51:16.708244 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:51:16.708244 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:51:16.708244 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:51:16.708244 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:51:16.708244 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:51:16.708244 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application with Lock( Oct 20 08:51:16.708244 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:51:16.708244 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:51:16.708244 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:51:16.708244 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:51:16.708244 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:51:16.708244 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:51:16.708244 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:51:16.708244 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:51:16.709327 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 20 08:51:16.709327 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 20 08:51:16.709327 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 20 08:51:16.709327 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 20 08:51:16.709327 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 20 08:51:16.709327 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 20 08:51:16.709327 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 20 08:51:16.709327 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: ERROR keystone.server.flask.application Oct 20 08:51:16.709327 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 475/962] 10.222.0.36 () {56 vars in 1111 bytes} [Thu Oct 20 08:51:16 2022] GET /identity/v3/projects/service => generated 88 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:51:16.714310 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba25422e-2a77-44a6-a5ff-cea62607135a None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:51:16.809236 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba25422e-2a77-44a6-a5ff-cea62607135a None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:51:16.841217 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36f8f21e-cae3-43de-b22e-8d0c50aa5ad5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:51:16.841793 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f8f21e-cae3-43de-b22e-8d0c50aa5ad5 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:16.842074 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f8f21e-cae3-43de-b22e-8d0c50aa5ad5 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:16.842340 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f8f21e-cae3-43de-b22e-8d0c50aa5ad5 None admin] PATH_INFO: `/v3/projects` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:16.843046 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36f8f21e-cae3-43de-b22e-8d0c50aa5ad5 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:51:16.843348 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36f8f21e-cae3-43de-b22e-8d0c50aa5ad5 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:51:16.844750 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36f8f21e-cae3-43de-b22e-8d0c50aa5ad5 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:51:16.857228 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 488/963] 10.222.0.36 () {56 vars in 1148 bytes} [Thu Oct 20 08:51:16 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:51:16.863188 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-770ea5aa-7567-4c12-b14e-a31cd801b6fa None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:51:16.953842 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:51:16.960987 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-770ea5aa-7567-4c12-b14e-a31cd801b6fa None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:51:16.992613 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78bc87ee-ca67-473b-b7bd-48054a0c2461 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:51:16.993190 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78bc87ee-ca67-473b-b7bd-48054a0c2461 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:16.993474 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78bc87ee-ca67-473b-b7bd-48054a0c2461 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:16.993743 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78bc87ee-ca67-473b-b7bd-48054a0c2461 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:16.994453 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78bc87ee-ca67-473b-b7bd-48054a0c2461 None admin] RBAC: Adding query filter params (role.id=d9d3bc62f6e8453488c74856ef333bb3, scope.project.id=d9857b610fb542969cfb5efbbdb8ce61, user.id=ad03ab76b9c44b1cb369475afa25eddd) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:51:16.994787 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78bc87ee-ca67-473b-b7bd-48054a0c2461 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:51:16.996204 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78bc87ee-ca67-473b-b7bd-48054a0c2461 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:51:17.001764 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 476/964] 10.222.0.36 () {56 vars in 1374 bytes} [Thu Oct 20 08:51:16 2022] GET /identity/v3/role_assignments?user.id=ad03ab76b9c44b1cb369475afa25eddd&scope.project.id=d9857b610fb542969cfb5efbbdb8ce61&role.id=d9d3bc62f6e8453488c74856ef333bb3 => generated 608 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:51:19.132856 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-761ecb83-d1b6-4c2b-bc65-a2578f67125a None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:19.133422 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-761ecb83-d1b6-4c2b-bc65-a2578f67125a None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:19.133422 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-761ecb83-d1b6-4c2b-bc65-a2578f67125a None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:19.135315 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 489/965] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:51:19 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:51:19.141618 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0091a122-d812-4ce9-9902-7a2e3cce9d19 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:19.142094 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0091a122-d812-4ce9-9902-7a2e3cce9d19 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:19.142234 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0091a122-d812-4ce9-9902-7a2e3cce9d19 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:19.161162 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-0091a122-d812-4ce9-9902-7a2e3cce9d19 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:51:19.162261 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-0091a122-d812-4ce9-9902-7a2e3cce9d19 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:51:19.229389 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 477/966] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:51:19 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 20 08:51:19.496345 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-761ecb83-d1b6-4c2b-bc65-a2578f67125a None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:51:19.526214 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:51:19.594446 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-761ecb83-d1b6-4c2b-bc65-a2578f67125a None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:51:19.626838 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e8343e-500c-4f9f-8be2-7926acdceb03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:51:19.627444 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e8343e-500c-4f9f-8be2-7926acdceb03 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:19.627719 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e8343e-500c-4f9f-8be2-7926acdceb03 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:19.627985 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e8343e-500c-4f9f-8be2-7926acdceb03 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:19.628557 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7e8343e-500c-4f9f-8be2-7926acdceb03 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:51:19.629907 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7e8343e-500c-4f9f-8be2-7926acdceb03 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:51:19.638323 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-e7e8343e-500c-4f9f-8be2-7926acdceb03 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Oct 20 08:51:19.638323 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:51:19.638323 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 20 08:51:19.638323 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 20 08:51:19.638323 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 20 08:51:19.638323 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 20 08:51:19.638323 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 20 08:51:19.638323 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 20 08:51:19.638323 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 20 08:51:19.638323 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:51:19.638323 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 20 08:51:19.638323 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:51:19.638323 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:51:19.638323 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:51:19.638323 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:51:19.638323 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:51:19.638323 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:51:19.639546 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:51:19.639546 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:51:19.639546 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:51:19.639546 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:51:19.639546 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:51:19.639546 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:51:19.639546 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 20 08:51:19.639546 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 20 08:51:19.639546 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 20 08:51:19.639546 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 20 08:51:19.639546 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:51:19.639546 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:51:19.639546 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:51:19.639546 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:51:19.639546 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:51:19.639546 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:51:19.640605 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:51:19.640605 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:51:19.640605 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:51:19.640605 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:51:19.640605 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:51:19.640605 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:51:19.640605 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:51:19.640605 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:51:19.640605 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:51:19.640605 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:51:19.640605 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 20 08:51:19.640605 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 20 08:51:19.640605 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 20 08:51:19.640605 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:51:19.640605 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 490/967] 10.222.0.36 () {56 vars in 1117 bytes} [Thu Oct 20 08:51:19 2022] GET /identity/v3/services/placement => generated 90 bytes in 144 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:51:19.645976 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0091a122-d812-4ce9-9902-7a2e3cce9d19 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:51:19.732210 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:51:19.741683 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0091a122-d812-4ce9-9902-7a2e3cce9d19 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:51:19.774451 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ee947f3-7c75-4a99-bfd3-82ab583c422d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:51:19.775010 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee947f3-7c75-4a99-bfd3-82ab583c422d None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:19.775285 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee947f3-7c75-4a99-bfd3-82ab583c422d None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:19.775554 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee947f3-7c75-4a99-bfd3-82ab583c422d None admin] PATH_INFO: `/v3/services` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:19.776168 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ee947f3-7c75-4a99-bfd3-82ab583c422d None admin] RBAC: Adding query filter params (name=placement) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:51:19.776522 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ee947f3-7c75-4a99-bfd3-82ab583c422d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:51:19.777916 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ee947f3-7c75-4a99-bfd3-82ab583c422d None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:51:19.782654 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 478/968] 10.222.0.36 () {56 vars in 1116 bytes} [Thu Oct 20 08:51:19 2022] GET /identity/v3/services?name=placement => generated 126 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:51:19.788835 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e8343e-500c-4f9f-8be2-7926acdceb03 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:51:19.878078 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:51:19.884044 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e8343e-500c-4f9f-8be2-7926acdceb03 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:51:19.916591 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7df53925-8898-4be2-b287-d516cc9537da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:51:19.917171 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df53925-8898-4be2-b287-d516cc9537da None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:19.917443 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df53925-8898-4be2-b287-d516cc9537da None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:19.917708 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df53925-8898-4be2-b287-d516cc9537da None admin] PATH_INFO: `/v3/services` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:19.918317 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7df53925-8898-4be2-b287-d516cc9537da None admin] RBAC: Adding query filter params (type=placement) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:51:19.918642 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7df53925-8898-4be2-b287-d516cc9537da None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:51:19.919991 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7df53925-8898-4be2-b287-d516cc9537da None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:51:19.924186 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 491/969] 10.222.0.36 () {56 vars in 1116 bytes} [Thu Oct 20 08:51:19 2022] GET /identity/v3/services?type=placement => generated 126 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:51:22.026918 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f4a7c8-64e2-46ed-a331-46ef0ed9bcfc None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:22.027544 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f4a7c8-64e2-46ed-a331-46ef0ed9bcfc None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:22.027544 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f4a7c8-64e2-46ed-a331-46ef0ed9bcfc None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:22.028797 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 479/970] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:51:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:51:22.034127 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b30c3c-1b31-4ec4-89e0-224b03122c6b None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:22.034596 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b30c3c-1b31-4ec4-89e0-224b03122c6b None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:22.034736 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b30c3c-1b31-4ec4-89e0-224b03122c6b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:22.052811 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-35b30c3c-1b31-4ec4-89e0-224b03122c6b None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:51:22.053967 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-35b30c3c-1b31-4ec4-89e0-224b03122c6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:51:22.117114 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 492/971] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:51:22 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 20 08:51:22.383471 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80f4a7c8-64e2-46ed-a331-46ef0ed9bcfc None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:51:22.482431 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80f4a7c8-64e2-46ed-a331-46ef0ed9bcfc None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:51:22.514761 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70cbedad-c453-4729-a535-5d810d36a659 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:51:22.515349 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70cbedad-c453-4729-a535-5d810d36a659 None admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:22.515622 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70cbedad-c453-4729-a535-5d810d36a659 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:22.515887 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70cbedad-c453-4729-a535-5d810d36a659 None admin] PATH_INFO: `/v3/services` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:22.516529 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70cbedad-c453-4729-a535-5d810d36a659 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:51:22.517876 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70cbedad-c453-4729-a535-5d810d36a659 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:51:22.532323 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 480/972] 10.222.0.36 () {60 vars in 1141 bytes} [Thu Oct 20 08:51:22 2022] POST /identity/v3/services => generated 246 bytes in 149 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 08:51:24.646632 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1507d110-7673-4970-8479-fd6fc3900747 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:24.647146 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1507d110-7673-4970-8479-fd6fc3900747 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:24.647146 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1507d110-7673-4970-8479-fd6fc3900747 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:24.648419 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 493/973] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:51:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:51:24.653675 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bce939e-8e9c-4e68-9da5-8f19a5d95384 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:24.654061 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bce939e-8e9c-4e68-9da5-8f19a5d95384 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:24.654218 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bce939e-8e9c-4e68-9da5-8f19a5d95384 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:24.672316 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-7bce939e-8e9c-4e68-9da5-8f19a5d95384 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:51:24.673478 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-7bce939e-8e9c-4e68-9da5-8f19a5d95384 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:51:24.682959 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:51:24.744429 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 481/974] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:51:24 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 20 08:51:25.007125 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1507d110-7673-4970-8479-fd6fc3900747 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:51:25.095132 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:51:25.107156 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1507d110-7673-4970-8479-fd6fc3900747 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:51:25.146207 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4050af4-4f45-482c-82b6-44317174eb1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:51:25.146940 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4050af4-4f45-482c-82b6-44317174eb1e None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:25.147285 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4050af4-4f45-482c-82b6-44317174eb1e None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:25.147556 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4050af4-4f45-482c-82b6-44317174eb1e None admin] PATH_INFO: `/v3/services/placement` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:25.148253 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4050af4-4f45-482c-82b6-44317174eb1e None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:51:25.149795 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4050af4-4f45-482c-82b6-44317174eb1e None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:51:25.159321 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-a4050af4-4f45-482c-82b6-44317174eb1e None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Oct 20 08:51:25.159321 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:51:25.159321 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 20 08:51:25.159321 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 20 08:51:25.159321 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 20 08:51:25.159321 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 20 08:51:25.159321 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 20 08:51:25.159321 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 20 08:51:25.159321 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 20 08:51:25.159321 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:51:25.159321 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 20 08:51:25.159321 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:51:25.159321 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:51:25.159321 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:51:25.159321 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:51:25.159321 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:51:25.159321 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:51:25.160626 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:51:25.160626 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:51:25.160626 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:51:25.160626 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:51:25.160626 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:51:25.160626 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:51:25.160626 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 20 08:51:25.160626 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 20 08:51:25.160626 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 20 08:51:25.160626 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 20 08:51:25.160626 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:51:25.160626 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:51:25.160626 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:51:25.160626 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:51:25.160626 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:51:25.160626 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:51:25.161672 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:51:25.161672 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:51:25.161672 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:51:25.161672 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:51:25.161672 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:51:25.161672 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:51:25.161672 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:51:25.161672 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:51:25.161672 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:51:25.161672 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:51:25.161672 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 20 08:51:25.161672 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 20 08:51:25.161672 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 20 08:51:25.161672 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:51:25.161672 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 494/975] 10.222.0.36 () {56 vars in 1117 bytes} [Thu Oct 20 08:51:25 2022] GET /identity/v3/services/placement => generated 90 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:51:25.166497 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bce939e-8e9c-4e68-9da5-8f19a5d95384 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:51:25.263562 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bce939e-8e9c-4e68-9da5-8f19a5d95384 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:51:25.295862 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20a47727-92c4-4c5c-a2d7-9a7bb4341747 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:51:25.296441 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a47727-92c4-4c5c-a2d7-9a7bb4341747 None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:25.296714 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a47727-92c4-4c5c-a2d7-9a7bb4341747 None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:25.296976 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a47727-92c4-4c5c-a2d7-9a7bb4341747 None admin] PATH_INFO: `/v3/services` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:25.297591 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20a47727-92c4-4c5c-a2d7-9a7bb4341747 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:51:25.297883 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20a47727-92c4-4c5c-a2d7-9a7bb4341747 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:51:25.299266 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20a47727-92c4-4c5c-a2d7-9a7bb4341747 None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:51:25.304790 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 482/976] 10.222.0.36 () {56 vars in 1116 bytes} [Thu Oct 20 08:51:25 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:51:25.310784 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4050af4-4f45-482c-82b6-44317174eb1e None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:51:25.406638 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4050af4-4f45-482c-82b6-44317174eb1e None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:51:25.437597 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48214cc5-bc06-408e-86bf-294e3bc28dbe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:51:25.438176 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48214cc5-bc06-408e-86bf-294e3bc28dbe None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:25.438449 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48214cc5-bc06-408e-86bf-294e3bc28dbe None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:25.438746 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48214cc5-bc06-408e-86bf-294e3bc28dbe None admin] PATH_INFO: `/v3/endpoints` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:25.439427 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48214cc5-bc06-408e-86bf-294e3bc28dbe None admin] RBAC: Adding query filter params (interface=public, service_id=5dfb16c62d264941a033e962ac59d20f, region_id=RegionOne) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:51:25.439725 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48214cc5-bc06-408e-86bf-294e3bc28dbe None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:51:25.441603 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48214cc5-bc06-408e-86bf-294e3bc28dbe None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:51:25.446352 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 495/977] 10.222.0.36 () {56 vars in 1251 bytes} [Thu Oct 20 08:51:25 2022] GET /identity/v3/endpoints?service_id=5dfb16c62d264941a033e962ac59d20f&interface=public®ion_id=RegionOne => generated 194 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:51:25.452110 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20a47727-92c4-4c5c-a2d7-9a7bb4341747 None admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:51:25.455765 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:51:25.548231 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20a47727-92c4-4c5c-a2d7-9a7bb4341747 None admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:51:25.580132 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24bad174-a760-450c-be2c-72cd47eca89b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:51:25.580716 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24bad174-a760-450c-be2c-72cd47eca89b None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:25.580985 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24bad174-a760-450c-be2c-72cd47eca89b None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:25.581797 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24bad174-a760-450c-be2c-72cd47eca89b None admin] PATH_INFO: `/v3/services` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:25.582385 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24bad174-a760-450c-be2c-72cd47eca89b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:51:25.583761 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24bad174-a760-450c-be2c-72cd47eca89b None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:51:25.590527 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 483/978] 10.222.0.36 () {56 vars in 1087 bytes} [Thu Oct 20 08:51:25 2022] GET /identity/v3/services => generated 2191 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 08:51:27.697280 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc28611c-538f-457d-99ab-f985615f6811 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:27.697776 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc28611c-538f-457d-99ab-f985615f6811 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:27.697776 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc28611c-538f-457d-99ab-f985615f6811 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:27.699111 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 496/979] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:51:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:51:27.704395 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b806b346-c636-4d37-91b3-eea66d83e7fc None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:27.704849 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b806b346-c636-4d37-91b3-eea66d83e7fc None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:27.704949 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b806b346-c636-4d37-91b3-eea66d83e7fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:27.723232 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-b806b346-c636-4d37-91b3-eea66d83e7fc None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:51:27.724387 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-b806b346-c636-4d37-91b3-eea66d83e7fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:51:27.783226 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:51:27.789475 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 484/980] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:51:27 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 20 08:51:28.061059 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc28611c-538f-457d-99ab-f985615f6811 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:51:28.080459 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:51:28.179917 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc28611c-538f-457d-99ab-f985615f6811 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:51:28.211499 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15f56c93-3509-434f-aa80-b819907c41f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:51:28.212110 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f56c93-3509-434f-aa80-b819907c41f9 None admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:28.212385 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f56c93-3509-434f-aa80-b819907c41f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:28.212650 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f56c93-3509-434f-aa80-b819907c41f9 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:28.213213 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15f56c93-3509-434f-aa80-b819907c41f9 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:51:28.214592 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15f56c93-3509-434f-aa80-b819907c41f9 None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:51:28.223530 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application [None req-15f56c93-3509-434f-aa80-b819907c41f9 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Oct 20 08:51:28.223530 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:51:28.223530 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 20 08:51:28.223530 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 20 08:51:28.223530 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 20 08:51:28.223530 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 20 08:51:28.223530 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 20 08:51:28.223530 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 20 08:51:28.223530 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 20 08:51:28.223530 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:51:28.223530 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 20 08:51:28.223530 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:51:28.223530 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 20 08:51:28.223530 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 20 08:51:28.223530 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 20 08:51:28.223530 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 20 08:51:28.223530 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 20 08:51:28.224711 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 20 08:51:28.224711 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 20 08:51:28.224711 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 20 08:51:28.224711 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 20 08:51:28.224711 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 20 08:51:28.224711 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 20 08:51:28.224711 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 20 08:51:28.224711 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 20 08:51:28.224711 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 20 08:51:28.224711 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 20 08:51:28.224711 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 20 08:51:28.224711 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 20 08:51:28.224711 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 20 08:51:28.224711 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 20 08:51:28.224711 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 20 08:51:28.224711 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.get_or_create( Oct 20 08:51:28.225576 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 20 08:51:28.225576 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application with Lock( Oct 20 08:51:28.225576 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 20 08:51:28.225576 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self._enter() Oct 20 08:51:28.225576 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 20 08:51:28.225576 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 20 08:51:28.225576 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 20 08:51:28.225576 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application return self.creator() Oct 20 08:51:28.225576 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 20 08:51:28.225576 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application created_value = creator( Oct 20 08:51:28.225576 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 20 08:51:28.225576 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 20 08:51:28.225576 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 20 08:51:28.225576 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: ERROR keystone.server.flask.application Oct 20 08:51:28.225576 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 497/981] 10.222.0.36 () {56 vars in 1117 bytes} [Thu Oct 20 08:51:28 2022] GET /identity/v3/services/placement => generated 90 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 20 08:51:28.230655 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b806b346-c636-4d37-91b3-eea66d83e7fc None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:51:28.324939 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b806b346-c636-4d37-91b3-eea66d83e7fc None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:51:28.357511 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3543570-ef44-40ff-8b09-2e26a64fab1f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:51:28.358097 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3543570-ef44-40ff-8b09-2e26a64fab1f None admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:28.358370 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3543570-ef44-40ff-8b09-2e26a64fab1f None admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:28.358671 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3543570-ef44-40ff-8b09-2e26a64fab1f None admin] PATH_INFO: `/v3/services` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:28.359285 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3543570-ef44-40ff-8b09-2e26a64fab1f None admin] RBAC: Adding query filter params (name=placement) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:51:28.359580 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3543570-ef44-40ff-8b09-2e26a64fab1f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:51:28.360925 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3543570-ef44-40ff-8b09-2e26a64fab1f None admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:51:28.366448 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 485/982] 10.222.0.36 () {56 vars in 1116 bytes} [Thu Oct 20 08:51:28 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:51:28.372149 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15f56c93-3509-434f-aa80-b819907c41f9 None admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:51:28.451559 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:51:28.471065 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15f56c93-3509-434f-aa80-b819907c41f9 None admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:51:28.503096 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fc14a19-02fd-4a15-96cc-3c117d0c30ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:51:28.504221 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc14a19-02fd-4a15-96cc-3c117d0c30ac None admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:51:28.504510 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc14a19-02fd-4a15-96cc-3c117d0c30ac None admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:51:28.504774 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc14a19-02fd-4a15-96cc-3c117d0c30ac None admin] PATH_INFO: `/v3/endpoints` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:51:28.505418 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fc14a19-02fd-4a15-96cc-3c117d0c30ac None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:51:28.506789 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fc14a19-02fd-4a15-96cc-3c117d0c30ac None admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:51:28.525733 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 498/983] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 08:51:28 2022] POST /identity/v3/endpoints => generated 331 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 08:51:38.363463 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=69 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:51:38.521453 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=30,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:52:32.323930 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff23ca9-68fe-484c-aeb7-219c357512c6 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:52:32.324817 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff23ca9-68fe-484c-aeb7-219c357512c6 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:52:32.325253 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff23ca9-68fe-484c-aeb7-219c357512c6 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:52:32.326870 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 486/984] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:52:32 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:52:32.333584 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1fa2255-412b-4c76-b274-273cf92123a2 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:52:32.334091 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1fa2255-412b-4c76-b274-273cf92123a2 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:52:32.334188 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1fa2255-412b-4c76-b274-273cf92123a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:52:32.451807 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-a1fa2255-412b-4c76-b274-273cf92123a2 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:52:32.453871 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-a1fa2255-412b-4c76-b274-273cf92123a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:52:32.583101 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 499/985] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:52:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 251 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:52:32.877904 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa19020-40dc-4bf6-9a7c-ec8bff5ca95d None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:52:32.878504 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa19020-40dc-4bf6-9a7c-ec8bff5ca95d None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:52:32.878931 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa19020-40dc-4bf6-9a7c-ec8bff5ca95d None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:52:32.880895 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 487/986] 10.222.0.36 () {54 vars in 964 bytes} [Thu Oct 20 08:52:32 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:52:32.887424 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf1c476-1dd8-43c4-bbd2-073c9237601f None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:52:32.887713 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf1c476-1dd8-43c4-bbd2-073c9237601f None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:52:32.887994 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf1c476-1dd8-43c4-bbd2-073c9237601f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:52:32.987077 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-0cf1c476-1dd8-43c4-bbd2-073c9237601f None None] MFA Rules not processed for user `794ca389002d41748d7925216ed964a5`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:52:32.988506 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-0cf1c476-1dd8-43c4-bbd2-073c9237601f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:52:33.013083 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:52:33.079308 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 500/987] 10.222.0.36 () {58 vars in 1061 bytes} [Thu Oct 20 08:52:32 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:52:33.088308 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffa19020-40dc-4bf6-9a7c-ec8bff5ca95d None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:52:33.089983 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-ffa19020-40dc-4bf6-9a7c-ec8bff5ca95d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:52:33.223488 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffa19020-40dc-4bf6-9a7c-ec8bff5ca95d None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:52:33.302782 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bea1b5a-0f99-41b4-9d7b-436e9a6472ea service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:52:33.303561 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bea1b5a-0f99-41b4-9d7b-436e9a6472ea service neutron] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:52:33.303940 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bea1b5a-0f99-41b4-9d7b-436e9a6472ea service neutron] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:52:33.304327 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bea1b5a-0f99-41b4-9d7b-436e9a6472ea service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:52:33.306018 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-0bea1b5a-0f99-41b4-9d7b-436e9a6472ea service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:52:33.365085 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:52:33.425675 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bea1b5a-0f99-41b4-9d7b-436e9a6472ea service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:52:33.427312 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bea1b5a-0f99-41b4-9d7b-436e9a6472ea service neutron] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:52:33.447979 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 488/988] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 08:52:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 360 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 08:52:35.895911 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af2e5a0-cb1d-403a-b21d-f6656de52e46 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:52:35.896466 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af2e5a0-cb1d-403a-b21d-f6656de52e46 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:52:35.896466 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af2e5a0-cb1d-403a-b21d-f6656de52e46 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:52:35.897859 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 501/989] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:52:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:52:35.902945 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa4e8d7e-454e-4041-8c02-07844fef49d2 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:52:35.903320 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa4e8d7e-454e-4041-8c02-07844fef49d2 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:52:35.903484 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa4e8d7e-454e-4041-8c02-07844fef49d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:52:35.972168 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-aa4e8d7e-454e-4041-8c02-07844fef49d2 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:52:35.973274 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-aa4e8d7e-454e-4041-8c02-07844fef49d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:52:36.065533 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 489/990] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:52:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:52:36.341134 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3af2e5a0-cb1d-403a-b21d-f6656de52e46 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:52:36.342697 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-3af2e5a0-cb1d-403a-b21d-f6656de52e46 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:52:36.446463 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3af2e5a0-cb1d-403a-b21d-f6656de52e46 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:52:36.515589 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:52:36.517862 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebaf156a-40d2-47a8-bd62-0ba59d033eaf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:52:36.518707 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebaf156a-40d2-47a8-bd62-0ba59d033eaf service neutron] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:52:36.519161 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebaf156a-40d2-47a8-bd62-0ba59d033eaf service neutron] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:52:36.519485 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebaf156a-40d2-47a8-bd62-0ba59d033eaf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:52:36.521233 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-ebaf156a-40d2-47a8-bd62-0ba59d033eaf service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:52:36.626994 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebaf156a-40d2-47a8-bd62-0ba59d033eaf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:52:36.628408 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebaf156a-40d2-47a8-bd62-0ba59d033eaf service neutron] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:52:36.646048 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 502/991] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 08:52:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 08:52:39.064820 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-650e9da0-be02-43bb-be81-b80f6707c881 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:52:39.065374 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-650e9da0-be02-43bb-be81-b80f6707c881 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:52:39.065374 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-650e9da0-be02-43bb-be81-b80f6707c881 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:52:39.066714 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 490/992] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:52:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:52:39.071798 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31734bdf-7904-41a2-9ae2-ac7ea7c399af None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:52:39.072172 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31734bdf-7904-41a2-9ae2-ac7ea7c399af None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:52:39.072353 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31734bdf-7904-41a2-9ae2-ac7ea7c399af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:52:39.139251 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-31734bdf-7904-41a2-9ae2-ac7ea7c399af None None] MFA Rules not processed for user `3cfc8055c0054f8f837edfcbaf0b0fe3`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:52:39.140453 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-31734bdf-7904-41a2-9ae2-ac7ea7c399af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:52:39.191422 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:52:39.235975 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 503/993] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:52:39 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:52:39.515386 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-650e9da0-be02-43bb-be81-b80f6707c881 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:52:39.517539 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-650e9da0-be02-43bb-be81-b80f6707c881 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:52:39.528144 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:52:39.619754 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-650e9da0-be02-43bb-be81-b80f6707c881 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:52:39.693395 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd492c45-da96-4856-acaa-e87ffa503f23 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:52:39.693918 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd492c45-da96-4856-acaa-e87ffa503f23 service neutron] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:52:39.694193 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd492c45-da96-4856-acaa-e87ffa503f23 service neutron] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:52:39.694459 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd492c45-da96-4856-acaa-e87ffa503f23 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:52:39.696065 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-cd492c45-da96-4856-acaa-e87ffa503f23 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:52:39.758321 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:52:39.803983 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd492c45-da96-4856-acaa-e87ffa503f23 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:52:39.805407 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd492c45-da96-4856-acaa-e87ffa503f23 service neutron] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:52:39.823953 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 491/994] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 08:52:39 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 08:52:42.487918 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c44b08-841a-49e1-ba34-1f4ee946bbbf None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:52:42.488437 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c44b08-841a-49e1-ba34-1f4ee946bbbf None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:52:42.488437 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c44b08-841a-49e1-ba34-1f4ee946bbbf None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:52:42.489849 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 504/995] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:52:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:52:42.494994 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d3a5520-3d9b-44db-819f-b91826b9fa96 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:52:42.495392 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d3a5520-3d9b-44db-819f-b91826b9fa96 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:52:42.495543 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d3a5520-3d9b-44db-819f-b91826b9fa96 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:52:42.560036 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-6d3a5520-3d9b-44db-819f-b91826b9fa96 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:52:42.561233 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-6d3a5520-3d9b-44db-819f-b91826b9fa96 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:52:42.655668 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 492/996] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:52:42 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:52:42.935279 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72c44b08-841a-49e1-ba34-1f4ee946bbbf None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:52:42.936808 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-72c44b08-841a-49e1-ba34-1f4ee946bbbf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:52:43.037103 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72c44b08-841a-49e1-ba34-1f4ee946bbbf None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:52:43.110065 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39a6a3e7-6a85-4180-b84e-ecd881b60815 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:52:43.111198 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a6a3e7-6a85-4180-b84e-ecd881b60815 service neutron] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:52:43.111475 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a6a3e7-6a85-4180-b84e-ecd881b60815 service neutron] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:52:43.111740 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a6a3e7-6a85-4180-b84e-ecd881b60815 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:52:43.113331 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-39a6a3e7-6a85-4180-b84e-ecd881b60815 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:52:43.120248 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:52:43.221587 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39a6a3e7-6a85-4180-b84e-ecd881b60815 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:52:43.222976 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39a6a3e7-6a85-4180-b84e-ecd881b60815 service neutron] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:52:43.240911 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 505/997] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 08:52:42 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 08:52:46.016952 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ffd40de-5866-42ad-bff8-a3c4a56c0eef None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:52:46.017810 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ffd40de-5866-42ad-bff8-a3c4a56c0eef None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:52:46.018085 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ffd40de-5866-42ad-bff8-a3c4a56c0eef None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:52:46.019684 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 493/998] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:52:46 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:52:46.025524 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19f51a7b-eca1-49ae-a1ae-a712aae10231 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:52:46.026072 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19f51a7b-eca1-49ae-a1ae-a712aae10231 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:52:46.026072 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19f51a7b-eca1-49ae-a1ae-a712aae10231 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:52:46.099967 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-19f51a7b-eca1-49ae-a1ae-a712aae10231 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:52:46.101313 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-19f51a7b-eca1-49ae-a1ae-a712aae10231 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:52:46.165914 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:52:46.208070 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 506/999] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:52:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:52:46.488146 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ffd40de-5866-42ad-bff8-a3c4a56c0eef None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:52:46.489660 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-8ffd40de-5866-42ad-bff8-a3c4a56c0eef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:52:46.509432 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:52:46.594188 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ffd40de-5866-42ad-bff8-a3c4a56c0eef None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:52:46.666301 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd74c51b-9048-44f7-b8e6-b189cf08d5be service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:52:46.666841 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd74c51b-9048-44f7-b8e6-b189cf08d5be service neutron] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:52:46.667133 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd74c51b-9048-44f7-b8e6-b189cf08d5be service neutron] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:52:46.667394 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd74c51b-9048-44f7-b8e6-b189cf08d5be service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:52:46.668954 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-bd74c51b-9048-44f7-b8e6-b189cf08d5be service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:52:46.739871 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:52:46.777402 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd74c51b-9048-44f7-b8e6-b189cf08d5be service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:52:46.778902 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd74c51b-9048-44f7-b8e6-b189cf08d5be service neutron] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:52:46.796455 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 494/1000] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 08:52:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 08:52:48.989188 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f297d55-d43c-4364-b3d5-5f935fb69661 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:52:48.989712 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f297d55-d43c-4364-b3d5-5f935fb69661 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:52:48.989712 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f297d55-d43c-4364-b3d5-5f935fb69661 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:52:48.991122 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 507/1001] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:52:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:52:48.996501 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04642e98-fbf8-4c92-aeff-ad89bd375f58 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:52:48.996931 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04642e98-fbf8-4c92-aeff-ad89bd375f58 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:52:48.997031 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04642e98-fbf8-4c92-aeff-ad89bd375f58 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:52:49.062884 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-04642e98-fbf8-4c92-aeff-ad89bd375f58 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:52:49.064885 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-04642e98-fbf8-4c92-aeff-ad89bd375f58 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:52:49.173514 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 495/1002] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:52:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:52:49.451711 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f297d55-d43c-4364-b3d5-5f935fb69661 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:52:49.453244 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-2f297d55-d43c-4364-b3d5-5f935fb69661 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:52:49.558270 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f297d55-d43c-4364-b3d5-5f935fb69661 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:52:49.631541 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-650fb0fa-d944-4f9b-b78a-e8cd27f46e06 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:52:49.632060 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-650fb0fa-d944-4f9b-b78a-e8cd27f46e06 service neutron] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:52:49.632339 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-650fb0fa-d944-4f9b-b78a-e8cd27f46e06 service neutron] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:52:49.632610 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-650fb0fa-d944-4f9b-b78a-e8cd27f46e06 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:52:49.634190 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-650fb0fa-d944-4f9b-b78a-e8cd27f46e06 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:52:49.651570 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:52:49.744814 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-650fb0fa-d944-4f9b-b78a-e8cd27f46e06 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:52:49.746249 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-650fb0fa-d944-4f9b-b78a-e8cd27f46e06 service neutron] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:52:49.764385 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 508/1003] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 08:52:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 08:52:51.973240 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e47d70-c2ec-4cf8-9245-03fbd60a44cc None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:52:51.973785 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e47d70-c2ec-4cf8-9245-03fbd60a44cc None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:52:51.973785 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e47d70-c2ec-4cf8-9245-03fbd60a44cc None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:52:51.975153 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 496/1004] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:52:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:52:51.980453 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10afdb87-3838-4b29-9602-1a8fe35180b2 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:52:51.980889 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10afdb87-3838-4b29-9602-1a8fe35180b2 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:52:51.981001 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10afdb87-3838-4b29-9602-1a8fe35180b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:52:52.053348 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-10afdb87-3838-4b29-9602-1a8fe35180b2 None None] MFA Rules not processed for user `3cfc8055c0054f8f837edfcbaf0b0fe3`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:52:52.054605 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-10afdb87-3838-4b29-9602-1a8fe35180b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:52:52.125065 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:52:52.162137 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 509/1005] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:52:51 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:52:52.461129 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab51ac8-4256-4392-a87b-623a62a9fc57 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:52:52.461408 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab51ac8-4256-4392-a87b-623a62a9fc57 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:52:52.461730 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab51ac8-4256-4392-a87b-623a62a9fc57 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:52:52.464003 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 497/1006] 10.222.0.36 () {54 vars in 964 bytes} [Thu Oct 20 08:52:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:52:52.469126 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c154304-ded0-47dc-93fe-0d7294bea5b0 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:52:52.469971 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c154304-ded0-47dc-93fe-0d7294bea5b0 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:52:52.470317 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c154304-ded0-47dc-93fe-0d7294bea5b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:52:52.546030 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-2c154304-ded0-47dc-93fe-0d7294bea5b0 None None] MFA Rules not processed for user `794ca389002d41748d7925216ed964a5`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:52:52.547158 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-2c154304-ded0-47dc-93fe-0d7294bea5b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:52:52.621494 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 510/1007] 10.222.0.36 () {58 vars in 1061 bytes} [Thu Oct 20 08:52:52 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 20 08:52:52.630666 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ab51ac8-4256-4392-a87b-623a62a9fc57 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:52:52.632180 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-6ab51ac8-4256-4392-a87b-623a62a9fc57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:52:52.663927 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:52:52.737608 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ab51ac8-4256-4392-a87b-623a62a9fc57 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:52:52.811472 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-857b2460-3b30-42b1-9d74-f3fb166525fe service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:52:52.811998 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857b2460-3b30-42b1-9d74-f3fb166525fe service neutron] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:52:52.812279 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857b2460-3b30-42b1-9d74-f3fb166525fe service neutron] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:52:52.812549 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857b2460-3b30-42b1-9d74-f3fb166525fe service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:52:52.814779 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-857b2460-3b30-42b1-9d74-f3fb166525fe service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:52:52.894731 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:52:52.927264 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-857b2460-3b30-42b1-9d74-f3fb166525fe service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:52:52.928716 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-857b2460-3b30-42b1-9d74-f3fb166525fe service neutron] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:52:52.950495 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 498/1008] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 08:52:52 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 320 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 08:52:55.690117 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61fa110f-ea0e-48df-84ae-9929244ce4e1 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:52:55.690702 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61fa110f-ea0e-48df-84ae-9929244ce4e1 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:52:55.690702 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61fa110f-ea0e-48df-84ae-9929244ce4e1 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:52:55.692018 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 511/1009] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:52:55 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:52:55.697277 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d426718-e13e-477a-8a8f-2b3bdd9d725f None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:52:55.697686 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d426718-e13e-477a-8a8f-2b3bdd9d725f None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:52:55.697819 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d426718-e13e-477a-8a8f-2b3bdd9d725f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:52:55.762248 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-5d426718-e13e-477a-8a8f-2b3bdd9d725f None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:52:55.763461 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-5d426718-e13e-477a-8a8f-2b3bdd9d725f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:52:55.857164 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 499/1010] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:52:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:52:56.139580 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61fa110f-ea0e-48df-84ae-9929244ce4e1 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:52:56.141169 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-61fa110f-ea0e-48df-84ae-9929244ce4e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:52:56.209794 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:52:56.244182 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61fa110f-ea0e-48df-84ae-9929244ce4e1 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:52:56.315460 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03bb813c-620d-4ea1-87d5-98957e097387 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:52:56.316057 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03bb813c-620d-4ea1-87d5-98957e097387 service neutron] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:52:56.316341 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03bb813c-620d-4ea1-87d5-98957e097387 service neutron] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:52:56.316612 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03bb813c-620d-4ea1-87d5-98957e097387 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:52:56.318204 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-03bb813c-620d-4ea1-87d5-98957e097387 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:52:56.425862 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03bb813c-620d-4ea1-87d5-98957e097387 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:52:56.427247 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03bb813c-620d-4ea1-87d5-98957e097387 service neutron] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:52:56.436895 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:52:56.445311 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 512/1011] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 08:52:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 20 08:52:58.618376 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db263087-2041-4018-9d17-74efe618ac35 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:52:58.618959 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db263087-2041-4018-9d17-74efe618ac35 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:52:58.618959 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db263087-2041-4018-9d17-74efe618ac35 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:52:58.620263 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 500/1012] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:52:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:52:58.625916 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c65afdfa-ddf9-43d2-9fba-6b9042d4578f None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:52:58.626321 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c65afdfa-ddf9-43d2-9fba-6b9042d4578f None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:52:58.626593 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c65afdfa-ddf9-43d2-9fba-6b9042d4578f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:52:58.692406 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-c65afdfa-ddf9-43d2-9fba-6b9042d4578f None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:52:58.693547 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-c65afdfa-ddf9-43d2-9fba-6b9042d4578f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:52:58.787600 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 513/1013] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:52:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:52:59.075210 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db263087-2041-4018-9d17-74efe618ac35 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:52:59.076759 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-db263087-2041-4018-9d17-74efe618ac35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:52:59.118878 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:52:59.180393 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db263087-2041-4018-9d17-74efe618ac35 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:52:59.252564 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bd10a2f-4e91-4851-9fba-00495b6fab52 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:52:59.253157 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bd10a2f-4e91-4851-9fba-00495b6fab52 service neutron] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:52:59.253432 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bd10a2f-4e91-4851-9fba-00495b6fab52 service neutron] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:52:59.253699 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bd10a2f-4e91-4851-9fba-00495b6fab52 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:52:59.255308 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-4bd10a2f-4e91-4851-9fba-00495b6fab52 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:52:59.337372 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:52:59.362952 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bd10a2f-4e91-4851-9fba-00495b6fab52 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:52:59.364397 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bd10a2f-4e91-4851-9fba-00495b6fab52 service neutron] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:52:59.383552 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 501/1014] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 08:52:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 08:53:01.934106 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-319e35d9-2dff-4927-a511-1a2e7d01f9a9 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:53:01.934679 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-319e35d9-2dff-4927-a511-1a2e7d01f9a9 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:53:01.934679 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-319e35d9-2dff-4927-a511-1a2e7d01f9a9 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:53:01.935996 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 514/1015] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:53:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:53:01.941573 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab35d1eb-cadb-4093-ba72-dab85ac57334 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:53:01.942007 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab35d1eb-cadb-4093-ba72-dab85ac57334 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:53:01.942112 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab35d1eb-cadb-4093-ba72-dab85ac57334 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:53:02.007347 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-ab35d1eb-cadb-4093-ba72-dab85ac57334 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:53:02.008461 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-ab35d1eb-cadb-4093-ba72-dab85ac57334 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:53:02.102737 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 502/1016] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:53:01 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:53:02.384803 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-319e35d9-2dff-4927-a511-1a2e7d01f9a9 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:53:02.386372 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-319e35d9-2dff-4927-a511-1a2e7d01f9a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:53:02.460961 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:53:02.488758 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-319e35d9-2dff-4927-a511-1a2e7d01f9a9 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:53:02.560642 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb0efc0c-3d81-49db-86e9-f2bcc4b143c7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:53:02.561237 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb0efc0c-3d81-49db-86e9-f2bcc4b143c7 service neutron] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:53:02.561512 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb0efc0c-3d81-49db-86e9-f2bcc4b143c7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:53:02.561781 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb0efc0c-3d81-49db-86e9-f2bcc4b143c7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:53:02.564014 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-eb0efc0c-3d81-49db-86e9-f2bcc4b143c7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:53:02.669572 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb0efc0c-3d81-49db-86e9-f2bcc4b143c7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:53:02.671020 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb0efc0c-3d81-49db-86e9-f2bcc4b143c7 service neutron] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:53:02.689097 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 515/1017] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 08:53:02 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 08:53:06.728865 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ddbb5b1-7782-460a-960c-54f1ae59382f None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:53:06.729383 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ddbb5b1-7782-460a-960c-54f1ae59382f None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:53:06.729383 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ddbb5b1-7782-460a-960c-54f1ae59382f None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:53:06.730733 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 503/1018] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:53:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:53:06.736133 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67196ed9-5ab9-43b5-86f9-a14d3c6c04cd None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:53:06.736544 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67196ed9-5ab9-43b5-86f9-a14d3c6c04cd None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:53:06.736681 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67196ed9-5ab9-43b5-86f9-a14d3c6c04cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:53:06.742209 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:53:06.805978 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-67196ed9-5ab9-43b5-86f9-a14d3c6c04cd None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:53:06.807138 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-67196ed9-5ab9-43b5-86f9-a14d3c6c04cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:53:06.901244 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 516/1019] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:53:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:53:07.185286 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ddbb5b1-7782-460a-960c-54f1ae59382f None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:53:07.186860 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-4ddbb5b1-7782-460a-960c-54f1ae59382f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:53:07.236171 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:53:07.291492 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ddbb5b1-7782-460a-960c-54f1ae59382f None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:53:07.365187 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60f380fc-8beb-4534-a2a2-f6b468fee9f0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:53:07.365923 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f380fc-8beb-4534-a2a2-f6b468fee9f0 service neutron] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:53:07.366191 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f380fc-8beb-4534-a2a2-f6b468fee9f0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:53:07.366459 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f380fc-8beb-4534-a2a2-f6b468fee9f0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:53:07.368072 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-60f380fc-8beb-4534-a2a2-f6b468fee9f0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:53:07.460967 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:53:07.478693 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60f380fc-8beb-4534-a2a2-f6b468fee9f0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:53:07.480128 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60f380fc-8beb-4534-a2a2-f6b468fee9f0 service neutron] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:53:07.499175 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 504/1020] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 08:53:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 08:53:10.357926 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba38c9ad-1d5a-4e05-8c45-d3531fd6edbb None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:53:10.358529 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba38c9ad-1d5a-4e05-8c45-d3531fd6edbb None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:53:10.358529 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba38c9ad-1d5a-4e05-8c45-d3531fd6edbb None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:53:10.360399 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 517/1021] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:53:10 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:53:10.365531 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-854813d3-69e6-48d8-ab81-0ea849ccbd58 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:53:10.365956 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-854813d3-69e6-48d8-ab81-0ea849ccbd58 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:53:10.366049 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-854813d3-69e6-48d8-ab81-0ea849ccbd58 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:53:10.433273 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-854813d3-69e6-48d8-ab81-0ea849ccbd58 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:53:10.434494 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-854813d3-69e6-48d8-ab81-0ea849ccbd58 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:53:10.536287 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 505/1022] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:53:10 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:53:10.822034 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba38c9ad-1d5a-4e05-8c45-d3531fd6edbb None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:53:10.823713 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-ba38c9ad-1d5a-4e05-8c45-d3531fd6edbb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:53:10.910798 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:53:10.928305 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba38c9ad-1d5a-4e05-8c45-d3531fd6edbb None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:53:11.002536 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce41e956-f6e1-4847-a844-6c2f7e1f07d2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:53:11.003059 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce41e956-f6e1-4847-a844-6c2f7e1f07d2 service neutron] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:53:11.003337 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce41e956-f6e1-4847-a844-6c2f7e1f07d2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:53:11.003607 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce41e956-f6e1-4847-a844-6c2f7e1f07d2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:53:11.005202 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-ce41e956-f6e1-4847-a844-6c2f7e1f07d2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:53:11.119155 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce41e956-f6e1-4847-a844-6c2f7e1f07d2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:53:11.121064 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce41e956-f6e1-4847-a844-6c2f7e1f07d2 service neutron] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:53:11.142651 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 518/1023] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 08:53:10 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 321 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 08:53:15.216326 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84393128-b703-47e4-af1b-a3c53b545038 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:53:15.216852 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84393128-b703-47e4-af1b-a3c53b545038 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:53:15.216852 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84393128-b703-47e4-af1b-a3c53b545038 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:53:15.218203 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 506/1024] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:53:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:53:15.223988 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fde05c60-5e26-4dec-acfb-be0d98cea13d None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:53:15.224409 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fde05c60-5e26-4dec-acfb-be0d98cea13d None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:53:15.224540 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fde05c60-5e26-4dec-acfb-be0d98cea13d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:53:15.240403 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:53:15.299136 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-fde05c60-5e26-4dec-acfb-be0d98cea13d None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:53:15.300359 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-fde05c60-5e26-4dec-acfb-be0d98cea13d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:53:15.397989 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 519/1025] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:53:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 20 08:53:15.680534 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84393128-b703-47e4-af1b-a3c53b545038 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:53:15.682057 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-84393128-b703-47e4-af1b-a3c53b545038 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:53:15.738847 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:53:15.786552 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84393128-b703-47e4-af1b-a3c53b545038 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:53:15.861972 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52e986ec-5c84-4f7f-89be-faba2b4850e5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:53:15.862605 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52e986ec-5c84-4f7f-89be-faba2b4850e5 service neutron] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:53:15.862878 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52e986ec-5c84-4f7f-89be-faba2b4850e5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:53:15.863141 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52e986ec-5c84-4f7f-89be-faba2b4850e5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:53:15.864749 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-52e986ec-5c84-4f7f-89be-faba2b4850e5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:53:15.977397 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:53:15.981896 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52e986ec-5c84-4f7f-89be-faba2b4850e5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:53:15.983498 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52e986ec-5c84-4f7f-89be-faba2b4850e5 service neutron] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:53:16.003247 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 507/1026] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 08:53:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 323 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 08:53:20.835866 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c871e96-5c9b-4f87-a499-ac243b0d3622 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:53:20.836326 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c871e96-5c9b-4f87-a499-ac243b0d3622 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:53:20.836641 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c871e96-5c9b-4f87-a499-ac243b0d3622 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:53:20.839348 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 520/1027] 10.222.0.36 () {54 vars in 921 bytes} [Thu Oct 20 08:53:20 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:53:20.844437 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-126be969-0f1f-4737-8cde-d616095e710b None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:53:20.844862 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-126be969-0f1f-4737-8cde-d616095e710b None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:53:20.845031 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-126be969-0f1f-4737-8cde-d616095e710b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:53:20.946925 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-126be969-0f1f-4737-8cde-d616095e710b None None] MFA Rules not processed for user `ad03ab76b9c44b1cb369475afa25eddd`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:53:20.948324 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-126be969-0f1f-4737-8cde-d616095e710b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:53:21.124086 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 508/1028] 10.222.0.36 () {58 vars in 1018 bytes} [Thu Oct 20 08:53:20 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 280 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:53:25.192632 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adfa421e-9e18-4735-8731-68a7317b72d5 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:53:25.193266 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adfa421e-9e18-4735-8731-68a7317b72d5 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:53:25.193266 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adfa421e-9e18-4735-8731-68a7317b72d5 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:53:25.194531 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 521/1029] 10.222.0.36 () {54 vars in 921 bytes} [Thu Oct 20 08:53:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:53:25.199490 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0bb64e0-3776-491a-9bed-02890438c93c None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:53:25.200292 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0bb64e0-3776-491a-9bed-02890438c93c None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:53:25.200807 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0bb64e0-3776-491a-9bed-02890438c93c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:53:25.257180 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:53:25.282717 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-b0bb64e0-3776-491a-9bed-02890438c93c None None] MFA Rules not processed for user `ad03ab76b9c44b1cb369475afa25eddd`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:53:25.284243 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-b0bb64e0-3776-491a-9bed-02890438c93c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:53:25.396346 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=59 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:53:25.409223 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 509/1030] 10.222.0.36 () {58 vars in 1018 bytes} [Thu Oct 20 08:53:25 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 211 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:53:26.853416 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84240f37-4897-446f-86a7-69775303aaa3 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:53:26.853932 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84240f37-4897-446f-86a7-69775303aaa3 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:53:26.854145 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84240f37-4897-446f-86a7-69775303aaa3 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:53:26.855616 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 522/1031] 10.222.0.36 () {54 vars in 921 bytes} [Thu Oct 20 08:53:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:53:26.861279 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9896d7-da03-42f3-afe5-6fcc9740f650 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:53:26.861702 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9896d7-da03-42f3-afe5-6fcc9740f650 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:53:26.861804 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9896d7-da03-42f3-afe5-6fcc9740f650 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:53:26.940658 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-ba9896d7-da03-42f3-afe5-6fcc9740f650 None None] MFA Rules not processed for user `ad03ab76b9c44b1cb369475afa25eddd`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:53:26.941878 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-ba9896d7-da03-42f3-afe5-6fcc9740f650 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:53:26.982978 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:53:27.047499 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 510/1032] 10.222.0.36 () {58 vars in 1018 bytes} [Thu Oct 20 08:53:26 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 20 08:53:30.556737 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f993cdd-d2f3-4fbc-91b6-b8b866107433 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:53:30.557305 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f993cdd-d2f3-4fbc-91b6-b8b866107433 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:53:30.557305 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f993cdd-d2f3-4fbc-91b6-b8b866107433 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:53:30.558630 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 523/1033] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:53:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:53:30.564014 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0500a8e2-880b-48d1-a8d7-c567532663f5 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:53:30.564424 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0500a8e2-880b-48d1-a8d7-c567532663f5 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:53:30.564538 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0500a8e2-880b-48d1-a8d7-c567532663f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:53:30.635519 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-0500a8e2-880b-48d1-a8d7-c567532663f5 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:53:30.636613 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-0500a8e2-880b-48d1-a8d7-c567532663f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:53:30.737635 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 511/1034] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:53:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:53:31.039972 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e29511-5611-4ef1-b6ba-c5f9a892fb3a None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:53:31.040449 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e29511-5611-4ef1-b6ba-c5f9a892fb3a None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:53:31.040910 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e29511-5611-4ef1-b6ba-c5f9a892fb3a None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:53:31.042405 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 524/1035] 10.222.0.36 () {54 vars in 961 bytes} [Thu Oct 20 08:53:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:53:31.050707 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f085aa39-0fe9-477c-875e-40079f5a5b2d None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:53:31.050995 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f085aa39-0fe9-477c-875e-40079f5a5b2d None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:53:31.051267 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f085aa39-0fe9-477c-875e-40079f5a5b2d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:53:31.072719 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:53:31.139128 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-f085aa39-0fe9-477c-875e-40079f5a5b2d None None] MFA Rules not processed for user `a94e868115ec49918eda10724263930c`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:53:31.141292 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-f085aa39-0fe9-477c-875e-40079f5a5b2d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:53:31.261291 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 512/1036] 10.222.0.36 () {58 vars in 1058 bytes} [Thu Oct 20 08:53:31 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 211 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:53:31.269375 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e29511-5611-4ef1-b6ba-c5f9a892fb3a None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:53:31.270895 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-f8e29511-5611-4ef1-b6ba-c5f9a892fb3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:53:31.430845 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e29511-5611-4ef1-b6ba-c5f9a892fb3a None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:53:31.508689 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:53:31.553468 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-511fd884-7cec-437d-8825-08a4fb056b64 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a94e868115ec49918eda10724263930c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:53:31.553992 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-511fd884-7cec-437d-8825-08a4fb056b64 service nova] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:53:31.554376 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-511fd884-7cec-437d-8825-08a4fb056b64 service nova] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:53:31.554674 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-511fd884-7cec-437d-8825-08a4fb056b64 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:53:31.556279 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-511fd884-7cec-437d-8825-08a4fb056b64 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:53:31.650976 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-511fd884-7cec-437d-8825-08a4fb056b64 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:53:31.652727 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-511fd884-7cec-437d-8825-08a4fb056b64 service nova] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:53:31.673886 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 525/1037] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 08:53:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 405 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 08:53:34.026327 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa619094-c58c-4899-b239-8bbdc5259448 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:53:34.026855 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa619094-c58c-4899-b239-8bbdc5259448 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:53:34.026855 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa619094-c58c-4899-b239-8bbdc5259448 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:53:34.028163 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 513/1038] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:53:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:53:34.033612 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-952f3ce1-69bd-430a-986f-a49c2ff67d33 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:53:34.034035 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-952f3ce1-69bd-430a-986f-a49c2ff67d33 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:53:34.034217 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-952f3ce1-69bd-430a-986f-a49c2ff67d33 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:53:34.132915 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-952f3ce1-69bd-430a-986f-a49c2ff67d33 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:53:34.134032 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-952f3ce1-69bd-430a-986f-a49c2ff67d33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:53:34.176867 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:53:34.245581 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 526/1039] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:53:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 213 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 20 08:53:34.529682 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa619094-c58c-4899-b239-8bbdc5259448 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:53:34.531756 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-aa619094-c58c-4899-b239-8bbdc5259448 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:53:34.602942 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:53:34.697492 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa619094-c58c-4899-b239-8bbdc5259448 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:53:34.830259 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab7a09ab-d9e7-4727-832c-afc74d4b6a51 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a94e868115ec49918eda10724263930c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:53:34.830899 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7a09ab-d9e7-4727-832c-afc74d4b6a51 service nova] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:53:34.831168 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7a09ab-d9e7-4727-832c-afc74d4b6a51 service nova] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:53:34.831429 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7a09ab-d9e7-4727-832c-afc74d4b6a51 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:53:34.832987 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-ab7a09ab-d9e7-4727-832c-afc74d4b6a51 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:53:34.864121 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:53:34.935993 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab7a09ab-d9e7-4727-832c-afc74d4b6a51 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:53:34.937574 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab7a09ab-d9e7-4727-832c-afc74d4b6a51 service nova] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:53:34.956736 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 514/1040] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 08:53:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 427 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 08:53:37.436174 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7099eaef-8f7e-43f7-8008-212e093ee882 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:53:37.437240 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7099eaef-8f7e-43f7-8008-212e093ee882 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:53:37.437595 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7099eaef-8f7e-43f7-8008-212e093ee882 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:53:37.439335 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 527/1041] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:53:37 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:53:37.445038 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5de7fc2-a7c4-4d30-8f4a-855af3f3db1f None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:53:37.445497 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5de7fc2-a7c4-4d30-8f4a-855af3f3db1f None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:53:37.445591 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5de7fc2-a7c4-4d30-8f4a-855af3f3db1f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:53:37.534527 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-a5de7fc2-a7c4-4d30-8f4a-855af3f3db1f None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:53:37.536141 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-a5de7fc2-a7c4-4d30-8f4a-855af3f3db1f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:53:37.646571 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 515/1042] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:53:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:53:37.649784 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:53:37.937479 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b121c256-2685-4601-be6a-2da2d656b27e None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:53:37.937998 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b121c256-2685-4601-be6a-2da2d656b27e None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:53:37.938550 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b121c256-2685-4601-be6a-2da2d656b27e None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:53:37.940446 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 528/1043] 10.222.0.36 () {54 vars in 961 bytes} [Thu Oct 20 08:53:37 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:53:37.946669 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4adca05a-25d4-4f8b-a5c0-6f0dd7e0309d None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:53:37.947528 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4adca05a-25d4-4f8b-a5c0-6f0dd7e0309d None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:53:37.948026 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4adca05a-25d4-4f8b-a5c0-6f0dd7e0309d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:53:38.026722 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-4adca05a-25d4-4f8b-a5c0-6f0dd7e0309d None None] MFA Rules not processed for user `a94e868115ec49918eda10724263930c`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:53:38.027899 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-4adca05a-25d4-4f8b-a5c0-6f0dd7e0309d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:53:38.151031 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 516/1044] 10.222.0.36 () {58 vars in 1058 bytes} [Thu Oct 20 08:53:37 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:53:38.159214 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b121c256-2685-4601-be6a-2da2d656b27e None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:53:38.160889 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-b121c256-2685-4601-be6a-2da2d656b27e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:53:38.319380 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b121c256-2685-4601-be6a-2da2d656b27e None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:53:38.341339 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:53:38.453113 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68d2537a-8809-47e4-8038-374d8329558a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a94e868115ec49918eda10724263930c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:53:38.453801 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d2537a-8809-47e4-8038-374d8329558a service nova] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:53:38.454038 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d2537a-8809-47e4-8038-374d8329558a service nova] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:53:38.454247 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d2537a-8809-47e4-8038-374d8329558a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:53:38.455905 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-68d2537a-8809-47e4-8038-374d8329558a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:53:38.560584 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68d2537a-8809-47e4-8038-374d8329558a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:53:38.562254 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68d2537a-8809-47e4-8038-374d8329558a service nova] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:53:38.583136 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 529/1045] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 08:53:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 424 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 08:53:38.713570 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c136d1-bc85-47b4-b293-d0d8fedf23f2 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:53:38.714123 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c136d1-bc85-47b4-b293-d0d8fedf23f2 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:53:38.714123 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c136d1-bc85-47b4-b293-d0d8fedf23f2 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:53:38.715802 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 517/1046] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:53:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:53:38.721823 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee85964-0668-4f5d-972e-c405df791f0e None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:53:38.722123 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee85964-0668-4f5d-972e-c405df791f0e None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:53:38.722399 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee85964-0668-4f5d-972e-c405df791f0e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:53:38.759410 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:53:38.802269 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-eee85964-0668-4f5d-972e-c405df791f0e None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:53:38.803534 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-eee85964-0668-4f5d-972e-c405df791f0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:53:38.926709 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 530/1047] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:53:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 206 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:53:39.275334 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b3d0cb-189e-49e3-adf6-0d6b440cdf7d None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:53:39.275690 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b3d0cb-189e-49e3-adf6-0d6b440cdf7d None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:53:39.276029 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b3d0cb-189e-49e3-adf6-0d6b440cdf7d None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:53:39.277580 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 518/1048] 10.222.0.36 () {54 vars in 963 bytes} [Thu Oct 20 08:53:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:53:39.282905 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c3a171-22c4-4f46-ae15-4181ea4463ed None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:53:39.283259 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c3a171-22c4-4f46-ae15-4181ea4463ed None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:53:39.283614 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c3a171-22c4-4f46-ae15-4181ea4463ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:53:39.369608 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-48c3a171-22c4-4f46-ae15-4181ea4463ed None None] MFA Rules not processed for user `7cfa6bb090de44eaa5bdf099a3096e8c`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:53:39.371132 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-48c3a171-22c4-4f46-ae15-4181ea4463ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:53:39.426329 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:53:39.463191 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 531/1049] 10.222.0.36 () {58 vars in 1060 bytes} [Thu Oct 20 08:53:39 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:53:39.471835 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7b3d0cb-189e-49e3-adf6-0d6b440cdf7d None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:53:39.473327 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-e7b3d0cb-189e-49e3-adf6-0d6b440cdf7d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:53:39.527842 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:53:39.588372 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7b3d0cb-189e-49e3-adf6-0d6b440cdf7d None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:53:39.671780 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65fb6651-9d67-4b9d-8a18-359e7d7c38d5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:53:39.672911 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65fb6651-9d67-4b9d-8a18-359e7d7c38d5 service cinder] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:53:39.673551 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65fb6651-9d67-4b9d-8a18-359e7d7c38d5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:53:39.674249 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65fb6651-9d67-4b9d-8a18-359e7d7c38d5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:53:39.676795 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-65fb6651-9d67-4b9d-8a18-359e7d7c38d5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:53:39.799276 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:53:39.813901 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65fb6651-9d67-4b9d-8a18-359e7d7c38d5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:53:39.815866 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65fb6651-9d67-4b9d-8a18-359e7d7c38d5 service cinder] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:53:39.842966 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 519/1050] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 08:53:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 20 08:53:40.808562 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb378748-3da5-4da0-b61a-36ffaf677700 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:53:40.809096 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb378748-3da5-4da0-b61a-36ffaf677700 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:53:40.809096 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb378748-3da5-4da0-b61a-36ffaf677700 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:53:40.810365 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 532/1051] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:53:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:53:40.816296 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e59ba1-df93-455b-9802-34dbc5721710 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:53:40.816712 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e59ba1-df93-455b-9802-34dbc5721710 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:53:40.816846 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e59ba1-df93-455b-9802-34dbc5721710 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:53:40.883346 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-88e59ba1-df93-455b-9802-34dbc5721710 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:53:40.884555 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-88e59ba1-df93-455b-9802-34dbc5721710 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:53:40.982220 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 520/1052] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:53:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:53:41.274174 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb378748-3da5-4da0-b61a-36ffaf677700 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:53:41.275762 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-bb378748-3da5-4da0-b61a-36ffaf677700 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:53:41.439586 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb378748-3da5-4da0-b61a-36ffaf677700 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:53:41.493530 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:53:41.556376 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-243f6164-c314-4547-b07d-6174ed17868a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a94e868115ec49918eda10724263930c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:53:41.556972 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243f6164-c314-4547-b07d-6174ed17868a service nova] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:53:41.557243 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243f6164-c314-4547-b07d-6174ed17868a service nova] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:53:41.557508 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243f6164-c314-4547-b07d-6174ed17868a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:53:41.559101 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-243f6164-c314-4547-b07d-6174ed17868a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:53:41.651706 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-243f6164-c314-4547-b07d-6174ed17868a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:53:41.653296 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-243f6164-c314-4547-b07d-6174ed17868a service nova] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:53:41.672613 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 533/1053] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 08:53:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 398 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 08:53:43.888452 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec8e5641-6d0d-4efe-b00b-5a2cd82ba7a1 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:53:43.889118 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec8e5641-6d0d-4efe-b00b-5a2cd82ba7a1 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:53:43.889118 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec8e5641-6d0d-4efe-b00b-5a2cd82ba7a1 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:53:43.890580 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 521/1054] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:53:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:53:43.895736 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78c5a32d-0c8f-4fcf-8877-5c418992a7ed None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:53:43.896187 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78c5a32d-0c8f-4fcf-8877-5c418992a7ed None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:53:43.896286 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78c5a32d-0c8f-4fcf-8877-5c418992a7ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:53:43.964457 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-78c5a32d-0c8f-4fcf-8877-5c418992a7ed None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:53:43.966299 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-78c5a32d-0c8f-4fcf-8877-5c418992a7ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:53:43.974448 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:53:44.071701 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 534/1055] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:53:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:53:44.366525 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec8e5641-6d0d-4efe-b00b-5a2cd82ba7a1 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:53:44.367726 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-ec8e5641-6d0d-4efe-b00b-5a2cd82ba7a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:53:44.431237 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:53:44.522101 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec8e5641-6d0d-4efe-b00b-5a2cd82ba7a1 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:53:44.638906 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1e1815a-900c-4f8f-9d60-d547d43df121 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a94e868115ec49918eda10724263930c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:53:44.640027 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e1815a-900c-4f8f-9d60-d547d43df121 service nova] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:53:44.640299 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e1815a-900c-4f8f-9d60-d547d43df121 service nova] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:53:44.640565 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e1815a-900c-4f8f-9d60-d547d43df121 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:53:44.642106 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-e1e1815a-900c-4f8f-9d60-d547d43df121 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:53:44.661850 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:53:44.735840 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1e1815a-900c-4f8f-9d60-d547d43df121 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:53:44.737463 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1e1815a-900c-4f8f-9d60-d547d43df121 service nova] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:53:44.756016 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 522/1056] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 08:53:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 390 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 08:53:46.949346 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25638b84-30d4-4b57-bbfc-22fd2b253c82 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:53:46.949935 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25638b84-30d4-4b57-bbfc-22fd2b253c82 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:53:46.950197 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25638b84-30d4-4b57-bbfc-22fd2b253c82 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:53:46.951864 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 535/1057] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:53:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:53:46.957077 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dbe7bb9-25f8-4cce-953c-6f6fe3fa45ea None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:53:46.957494 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dbe7bb9-25f8-4cce-953c-6f6fe3fa45ea None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:53:46.957675 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dbe7bb9-25f8-4cce-953c-6f6fe3fa45ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:53:47.025591 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-8dbe7bb9-25f8-4cce-953c-6f6fe3fa45ea None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:53:47.026816 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-8dbe7bb9-25f8-4cce-953c-6f6fe3fa45ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:53:47.132729 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:53:47.141326 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 523/1058] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:53:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:53:47.431588 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25638b84-30d4-4b57-bbfc-22fd2b253c82 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:53:47.434648 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-25638b84-30d4-4b57-bbfc-22fd2b253c82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:53:47.602550 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:53:47.613618 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25638b84-30d4-4b57-bbfc-22fd2b253c82 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:53:47.742981 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6d5caaf-4ca1-4912-b470-67de4c18ad6f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a94e868115ec49918eda10724263930c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:53:47.744062 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d5caaf-4ca1-4912-b470-67de4c18ad6f service nova] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:53:47.744590 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d5caaf-4ca1-4912-b470-67de4c18ad6f service nova] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:53:47.745161 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d5caaf-4ca1-4912-b470-67de4c18ad6f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:53:47.748823 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-f6d5caaf-4ca1-4912-b470-67de4c18ad6f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:53:47.895372 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6d5caaf-4ca1-4912-b470-67de4c18ad6f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:53:47.896970 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6d5caaf-4ca1-4912-b470-67de4c18ad6f service nova] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:53:47.907759 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:53:47.916690 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 536/1059] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 08:53:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 486 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 08:53:50.189918 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac8f49b-fd6d-4a14-b1b1-c89cbde46646 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:53:50.190459 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac8f49b-fd6d-4a14-b1b1-c89cbde46646 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:53:50.190459 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac8f49b-fd6d-4a14-b1b1-c89cbde46646 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:53:50.191854 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 524/1060] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:53:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:53:50.197130 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a7917b-0028-4bbd-92f1-dbf6dd2eb572 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:53:50.197556 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a7917b-0028-4bbd-92f1-dbf6dd2eb572 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:53:50.197694 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a7917b-0028-4bbd-92f1-dbf6dd2eb572 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:53:50.267103 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-c6a7917b-0028-4bbd-92f1-dbf6dd2eb572 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:53:50.268360 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-c6a7917b-0028-4bbd-92f1-dbf6dd2eb572 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:53:50.387099 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 537/1061] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:53:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:53:50.688426 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bac8f49b-fd6d-4a14-b1b1-c89cbde46646 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:53:50.689968 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-bac8f49b-fd6d-4a14-b1b1-c89cbde46646 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:53:50.843287 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bac8f49b-fd6d-4a14-b1b1-c89cbde46646 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:53:50.920270 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:53:50.965402 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-013e68d9-a705-42a6-ba75-bb4e25a083ae service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a94e868115ec49918eda10724263930c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:53:50.965969 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-013e68d9-a705-42a6-ba75-bb4e25a083ae service nova] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:53:50.966242 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-013e68d9-a705-42a6-ba75-bb4e25a083ae service nova] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:53:50.966536 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-013e68d9-a705-42a6-ba75-bb4e25a083ae service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:53:50.968162 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-013e68d9-a705-42a6-ba75-bb4e25a083ae service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:53:51.072407 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-013e68d9-a705-42a6-ba75-bb4e25a083ae service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:53:51.075586 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-013e68d9-a705-42a6-ba75-bb4e25a083ae service nova] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:53:51.106594 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 525/1062] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 08:53:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 419 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 08:53:53.271233 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f60acfb1-0d7d-4c8a-8c76-292f74baaaa9 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:53:53.271801 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f60acfb1-0d7d-4c8a-8c76-292f74baaaa9 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:53:53.271801 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f60acfb1-0d7d-4c8a-8c76-292f74baaaa9 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:53:53.273132 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 538/1063] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:53:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:53:53.278339 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b18f0f1-61f4-48e7-a681-0dccd8a919b8 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:53:53.278813 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b18f0f1-61f4-48e7-a681-0dccd8a919b8 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:53:53.278936 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b18f0f1-61f4-48e7-a681-0dccd8a919b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:53:53.346108 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-3b18f0f1-61f4-48e7-a681-0dccd8a919b8 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:53:53.347338 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-3b18f0f1-61f4-48e7-a681-0dccd8a919b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:53:53.385927 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:53:53.451083 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 526/1064] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:53:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:53:53.743950 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f60acfb1-0d7d-4c8a-8c76-292f74baaaa9 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:53:53.745498 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-f60acfb1-0d7d-4c8a-8c76-292f74baaaa9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:53:53.822838 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:53:53.893281 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f60acfb1-0d7d-4c8a-8c76-292f74baaaa9 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:53:54.004390 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a3dce67-2d62-41a8-8619-55383663bb64 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a94e868115ec49918eda10724263930c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:53:54.004899 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a3dce67-2d62-41a8-8619-55383663bb64 service nova] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:53:54.005175 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a3dce67-2d62-41a8-8619-55383663bb64 service nova] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:53:54.005441 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a3dce67-2d62-41a8-8619-55383663bb64 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:53:54.007040 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-8a3dce67-2d62-41a8-8619-55383663bb64 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:53:54.048252 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:53:54.100177 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a3dce67-2d62-41a8-8619-55383663bb64 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:53:54.101900 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a3dce67-2d62-41a8-8619-55383663bb64 service nova] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:53:54.119565 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 539/1065] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 08:53:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 376 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 08:53:55.003229 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14752540-a136-416d-976d-88eac6d79185 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:53:55.003798 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14752540-a136-416d-976d-88eac6d79185 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:53:55.003798 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14752540-a136-416d-976d-88eac6d79185 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:53:55.005176 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 527/1066] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:53:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:53:55.010465 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e114044e-8978-44b3-9ed9-a347be3034e2 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:53:55.010925 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e114044e-8978-44b3-9ed9-a347be3034e2 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:53:55.011042 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e114044e-8978-44b3-9ed9-a347be3034e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:53:55.078298 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-e114044e-8978-44b3-9ed9-a347be3034e2 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:53:55.079529 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-e114044e-8978-44b3-9ed9-a347be3034e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:53:55.177246 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 540/1067] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:53:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:53:55.482646 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77c9f454-4bdd-435f-95db-de3a5ed926ad None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:53:55.483320 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77c9f454-4bdd-435f-95db-de3a5ed926ad None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:53:55.483478 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77c9f454-4bdd-435f-95db-de3a5ed926ad None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:53:55.484952 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 528/1068] 10.222.0.36 () {54 vars in 962 bytes} [Thu Oct 20 08:53:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:53:55.490313 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5024da-e5de-4196-9a79-1d5b2e26b278 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:53:55.490643 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5024da-e5de-4196-9a79-1d5b2e26b278 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:53:55.490921 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5024da-e5de-4196-9a79-1d5b2e26b278 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:53:55.507081 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:53:55.569716 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-cb5024da-e5de-4196-9a79-1d5b2e26b278 None None] MFA Rules not processed for user `ce043914cd9042909f70ab0adb272d34`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:53:55.570937 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-cb5024da-e5de-4196-9a79-1d5b2e26b278 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:53:55.647039 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 541/1069] 10.222.0.36 () {58 vars in 1059 bytes} [Thu Oct 20 08:53:55 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:53:55.655728 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77c9f454-4bdd-435f-95db-de3a5ed926ad None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:53:55.657233 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-77c9f454-4bdd-435f-95db-de3a5ed926ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:53:55.762890 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77c9f454-4bdd-435f-95db-de3a5ed926ad None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:53:55.823031 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:53:55.839384 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c714499a-2f2e-4e2f-bf8d-62afdce8dc40 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce043914cd9042909f70ab0adb272d34', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:53:55.839984 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c714499a-2f2e-4e2f-bf8d-62afdce8dc40 service glance] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:53:55.840262 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c714499a-2f2e-4e2f-bf8d-62afdce8dc40 service glance] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:53:55.840521 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c714499a-2f2e-4e2f-bf8d-62afdce8dc40 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:53:55.842141 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-c714499a-2f2e-4e2f-bf8d-62afdce8dc40 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:53:55.956205 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c714499a-2f2e-4e2f-bf8d-62afdce8dc40 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:53:55.957551 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c714499a-2f2e-4e2f-bf8d-62afdce8dc40 service glance] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:53:55.976346 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 529/1070] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 08:53:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 321 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 08:53:56.301349 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e14e4a-52d4-4a40-ad7f-a4ba5b3ce1c9 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:53:56.301906 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e14e4a-52d4-4a40-ad7f-a4ba5b3ce1c9 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:53:56.301906 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e14e4a-52d4-4a40-ad7f-a4ba5b3ce1c9 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:53:56.303365 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 542/1071] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:53:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:53:56.308420 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba589a7-bdc5-4446-a441-f8621e0e088b None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:53:56.308873 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba589a7-bdc5-4446-a441-f8621e0e088b None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:53:56.309163 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba589a7-bdc5-4446-a441-f8621e0e088b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:53:56.384465 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-0ba589a7-bdc5-4446-a441-f8621e0e088b None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:53:56.385762 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-0ba589a7-bdc5-4446-a441-f8621e0e088b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:53:56.429183 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:53:56.488580 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 530/1072] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:53:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:53:56.720610 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e339ff0d-8b64-4a02-b049-d77246416a93 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:53:56.720888 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e339ff0d-8b64-4a02-b049-d77246416a93 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:53:56.721157 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e339ff0d-8b64-4a02-b049-d77246416a93 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:53:56.786587 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba589a7-bdc5-4446-a441-f8621e0e088b None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:53:56.788251 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-0ba589a7-bdc5-4446-a441-f8621e0e088b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:53:56.789957 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-e339ff0d-8b64-4a02-b049-d77246416a93 None None] MFA Rules not processed for user `f4b0e9187ce74f7888e9e7094be5c824`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:53:56.791189 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-e339ff0d-8b64-4a02-b049-d77246416a93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:53:56.847198 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:53:56.880224 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 543/1073] 10.222.0.36 () {58 vars in 1009 bytes} [Thu Oct 20 08:53:56 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:53:56.905883 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba589a7-bdc5-4446-a441-f8621e0e088b None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:53:56.922422 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d81088-8a18-411f-bec3-ae385409c0df None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:53:56.922738 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d81088-8a18-411f-bec3-ae385409c0df None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:53:56.923005 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d81088-8a18-411f-bec3-ae385409c0df None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:53:56.924343 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 544/1074] 10.222.0.36 () {54 vars in 943 bytes} [Thu Oct 20 08:53:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 20 08:53:56.930412 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c60fe7-023e-4779-9e97-e3524548461b None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:53:56.930731 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c60fe7-023e-4779-9e97-e3524548461b None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:53:56.931169 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c60fe7-023e-4779-9e97-e3524548461b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:53:56.968069 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:53:57.008555 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-b9c60fe7-023e-4779-9e97-e3524548461b None None] MFA Rules not processed for user `4d558918cd8b4ebea4fe8765297304db`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:53:57.009747 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-b9c60fe7-023e-4779-9e97-e3524548461b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:53:57.027349 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-131c52b6-3d94-42f1-9e77-1c95fe6ecc89 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a94e868115ec49918eda10724263930c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:53:57.027956 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-131c52b6-3d94-42f1-9e77-1c95fe6ecc89 service nova] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:53:57.028226 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-131c52b6-3d94-42f1-9e77-1c95fe6ecc89 service nova] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:53:57.028487 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-131c52b6-3d94-42f1-9e77-1c95fe6ecc89 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:53:57.030109 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-131c52b6-3d94-42f1-9e77-1c95fe6ecc89 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:53:57.118648 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 545/1075] 10.222.0.36 () {58 vars in 1040 bytes} [Thu Oct 20 08:53:56 2022] POST /identity/v3/auth/tokens => generated 3304 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 20 08:53:57.120982 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-131c52b6-3d94-42f1-9e77-1c95fe6ecc89 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:53:57.122639 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-131c52b6-3d94-42f1-9e77-1c95fe6ecc89 service nova] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:53:57.128650 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9c60fe7-023e-4779-9e97-e3524548461b None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:53:57.137861 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9c60fe7-023e-4779-9e97-e3524548461b None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:53:57.143318 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 531/1076] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 08:53:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 357 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 20 08:53:57.165976 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:53:57.253283 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ed5f340-0ec7-4ac4-b6ff-b2cb3a37423d service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4d558918cd8b4ebea4fe8765297304db', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:53:57.253901 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed5f340-0ec7-4ac4-b6ff-b2cb3a37423d service swift] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:53:57.254173 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed5f340-0ec7-4ac4-b6ff-b2cb3a37423d service swift] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:53:57.254439 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed5f340-0ec7-4ac4-b6ff-b2cb3a37423d service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:53:57.256146 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-9ed5f340-0ec7-4ac4-b6ff-b2cb3a37423d service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:53:57.351418 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ed5f340-0ec7-4ac4-b6ff-b2cb3a37423d service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:53:57.353024 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ed5f340-0ec7-4ac4-b6ff-b2cb3a37423d service swift] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:53:57.361978 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 546/1077] 10.222.0.36 () {60 vars in 1384 bytes} [Thu Oct 20 08:53:57 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 234 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 20 08:53:59.354070 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1611ec-b7e2-4886-9490-bd0d7f356739 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:53:59.354630 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1611ec-b7e2-4886-9490-bd0d7f356739 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:53:59.354752 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1611ec-b7e2-4886-9490-bd0d7f356739 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:53:59.356114 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 532/1078] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:53:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:53:59.362428 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35c1cb4-ba60-436e-9fb7-b9731236d571 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:53:59.362924 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35c1cb4-ba60-436e-9fb7-b9731236d571 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:53:59.363022 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35c1cb4-ba60-436e-9fb7-b9731236d571 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:53:59.432232 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-a35c1cb4-ba60-436e-9fb7-b9731236d571 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:53:59.433448 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-a35c1cb4-ba60-436e-9fb7-b9731236d571 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:53:59.441389 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:53:59.543380 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 547/1079] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:53:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:53:59.836405 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1611ec-b7e2-4886-9490-bd0d7f356739 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:53:59.837905 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-8c1611ec-b7e2-4886-9490-bd0d7f356739 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:53:59.913451 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:54:00.004733 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1611ec-b7e2-4886-9490-bd0d7f356739 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:54:00.191383 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-898e6cde-5cc2-4366-b770-65d21047574c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a94e868115ec49918eda10724263930c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:54:00.192159 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898e6cde-5cc2-4366-b770-65d21047574c service nova] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:54:00.192558 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898e6cde-5cc2-4366-b770-65d21047574c service nova] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:54:00.192835 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898e6cde-5cc2-4366-b770-65d21047574c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:54:00.194574 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-898e6cde-5cc2-4366-b770-65d21047574c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:54:00.230218 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:54:00.299741 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-898e6cde-5cc2-4366-b770-65d21047574c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:54:00.301443 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-898e6cde-5cc2-4366-b770-65d21047574c service nova] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:54:00.320319 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 533/1080] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 08:53:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 484 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 08:54:02.533511 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d66752-02ca-4514-970c-0054193664de None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:54:02.534078 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d66752-02ca-4514-970c-0054193664de None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:54:02.534078 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d66752-02ca-4514-970c-0054193664de None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:54:02.535583 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 548/1081] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:54:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:54:02.543227 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33966ea7-ed62-46af-9e15-219c492e9e7d None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:54:02.543227 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33966ea7-ed62-46af-9e15-219c492e9e7d None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:54:02.543227 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33966ea7-ed62-46af-9e15-219c492e9e7d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:54:02.613153 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-33966ea7-ed62-46af-9e15-219c492e9e7d None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:54:02.615599 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-33966ea7-ed62-46af-9e15-219c492e9e7d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:54:02.728907 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 534/1082] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:54:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:54:03.022183 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45d66752-02ca-4514-970c-0054193664de None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:54:03.024040 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-45d66752-02ca-4514-970c-0054193664de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:54:03.171405 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:54:03.182341 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45d66752-02ca-4514-970c-0054193664de None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:54:03.310124 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8b0e878-4c32-48f0-a85d-225754d79fe9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a94e868115ec49918eda10724263930c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:54:03.311271 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8b0e878-4c32-48f0-a85d-225754d79fe9 service nova] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:54:03.311549 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8b0e878-4c32-48f0-a85d-225754d79fe9 service nova] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:54:03.311826 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8b0e878-4c32-48f0-a85d-225754d79fe9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:54:03.313429 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-b8b0e878-4c32-48f0-a85d-225754d79fe9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:54:03.417214 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8b0e878-4c32-48f0-a85d-225754d79fe9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:54:03.419194 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8b0e878-4c32-48f0-a85d-225754d79fe9 service nova] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:54:03.430255 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:54:03.439360 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 549/1083] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 08:54:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 418 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 08:54:05.656367 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc8ee76-5bde-43b6-91df-e0f29eee5ce0 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:54:05.656887 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc8ee76-5bde-43b6-91df-e0f29eee5ce0 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:54:05.656887 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc8ee76-5bde-43b6-91df-e0f29eee5ce0 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:54:05.658166 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 535/1084] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:54:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:54:05.663431 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f5d366-e878-449e-a5d6-aedad7f3e42e None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:54:05.663841 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f5d366-e878-449e-a5d6-aedad7f3e42e None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:54:05.664026 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f5d366-e878-449e-a5d6-aedad7f3e42e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:54:05.738049 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-02f5d366-e878-449e-a5d6-aedad7f3e42e None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:54:05.739316 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-02f5d366-e878-449e-a5d6-aedad7f3e42e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:54:05.847155 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 550/1085] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:54:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:54:06.137141 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bc8ee76-5bde-43b6-91df-e0f29eee5ce0 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:54:06.139259 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-5bc8ee76-5bde-43b6-91df-e0f29eee5ce0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:54:06.142128 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:54:06.296194 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bc8ee76-5bde-43b6-91df-e0f29eee5ce0 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:54:06.374284 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:54:06.420463 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c300d85-f0c0-4857-a1bb-a4b58b6fe073 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a94e868115ec49918eda10724263930c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:54:06.421068 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c300d85-f0c0-4857-a1bb-a4b58b6fe073 service nova] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:54:06.421340 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c300d85-f0c0-4857-a1bb-a4b58b6fe073 service nova] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:54:06.421600 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c300d85-f0c0-4857-a1bb-a4b58b6fe073 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:54:06.423218 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-6c300d85-f0c0-4857-a1bb-a4b58b6fe073 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:54:06.519899 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c300d85-f0c0-4857-a1bb-a4b58b6fe073 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:54:06.522025 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c300d85-f0c0-4857-a1bb-a4b58b6fe073 service nova] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:54:06.541331 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 536/1086] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 08:54:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 405 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 08:54:15.845385 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=67 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:54:16.539313 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=61 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:54:19.693470 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c5c2930-228e-4ac4-9d7b-d3efab94c459 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:54:19.693984 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c5c2930-228e-4ac4-9d7b-d3efab94c459 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:54:19.693984 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c5c2930-228e-4ac4-9d7b-d3efab94c459 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:54:19.695321 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 551/1087] 10.222.0.36 () {54 vars in 918 bytes} [Thu Oct 20 08:54:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:54:19.701054 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0117d100-8cad-4821-96f4-69b65a326911 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:54:19.701510 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0117d100-8cad-4821-96f4-69b65a326911 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:54:19.701614 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0117d100-8cad-4821-96f4-69b65a326911 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:54:19.777041 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-0117d100-8cad-4821-96f4-69b65a326911 None None] MFA Rules not processed for user `ad03ab76b9c44b1cb369475afa25eddd`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:54:19.778859 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-0117d100-8cad-4821-96f4-69b65a326911 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:54:19.884816 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 537/1088] 10.222.0.36 () {58 vars in 1015 bytes} [Thu Oct 20 08:54:19 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:54:19.898774 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-701123f9-0efc-48d4-a2c5-87831dba4a72 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:54:19.899087 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-701123f9-0efc-48d4-a2c5-87831dba4a72 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:54:19.899362 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-701123f9-0efc-48d4-a2c5-87831dba4a72 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:54:19.900689 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 552/1089] 10.222.0.36 () {54 vars in 918 bytes} [Thu Oct 20 08:54:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:54:19.905650 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd21698-4196-4fc9-84f9-e9ed2ead1d21 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:54:19.905960 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd21698-4196-4fc9-84f9-e9ed2ead1d21 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:54:19.906231 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd21698-4196-4fc9-84f9-e9ed2ead1d21 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:54:19.978586 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:54:19.981865 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-4dd21698-4196-4fc9-84f9-e9ed2ead1d21 None None] MFA Rules not processed for user `a94e868115ec49918eda10724263930c`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:54:19.983565 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-4dd21698-4196-4fc9-84f9-e9ed2ead1d21 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:54:20.096935 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 538/1090] 10.222.0.36 () {58 vars in 1015 bytes} [Thu Oct 20 08:54:19 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:54:22.664787 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1888cb1c-789e-4c3d-9b92-f40a8043e5b6 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:54:22.665317 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1888cb1c-789e-4c3d-9b92-f40a8043e5b6 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:54:22.665317 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1888cb1c-789e-4c3d-9b92-f40a8043e5b6 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:54:22.666612 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 553/1091] 127.0.0.1 () {54 vars in 917 bytes} [Thu Oct 20 08:54:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:54:22.672038 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a952e6-97f8-443c-bb95-5b133832f2ce None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:54:22.672488 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a952e6-97f8-443c-bb95-5b133832f2ce None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:54:22.672597 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a952e6-97f8-443c-bb95-5b133832f2ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:54:22.738371 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-41a952e6-97f8-443c-bb95-5b133832f2ce None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:54:22.739576 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-41a952e6-97f8-443c-bb95-5b133832f2ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:54:22.794331 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:54:22.834777 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 539/1092] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Oct 20 08:54:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 20 08:54:23.127855 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1888cb1c-789e-4c3d-9b92-f40a8043e5b6 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:54:23.129626 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-1888cb1c-789e-4c3d-9b92-f40a8043e5b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:54:23.274175 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1888cb1c-789e-4c3d-9b92-f40a8043e5b6 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:54:23.346601 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:54:23.385840 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e917c7-1ee8-4240-be8e-78c5a430d35f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a94e868115ec49918eda10724263930c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:54:23.386450 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e917c7-1ee8-4240-be8e-78c5a430d35f service nova] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:54:23.386768 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e917c7-1ee8-4240-be8e-78c5a430d35f service nova] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:54:23.387033 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e917c7-1ee8-4240-be8e-78c5a430d35f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:54:23.389124 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-57e917c7-1ee8-4240-be8e-78c5a430d35f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:54:23.477678 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57e917c7-1ee8-4240-be8e-78c5a430d35f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:54:23.479286 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57e917c7-1ee8-4240-be8e-78c5a430d35f service nova] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:54:23.496450 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 554/1093] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 08:54:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 369 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 08:54:24.629294 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb85b304-3326-4d1d-aee3-2053b03ec204 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:54:24.629857 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb85b304-3326-4d1d-aee3-2053b03ec204 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:54:24.629857 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb85b304-3326-4d1d-aee3-2053b03ec204 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:54:24.631240 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 540/1094] 127.0.0.1 () {54 vars in 901 bytes} [Thu Oct 20 08:54:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:54:24.637987 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07eafc8e-8015-43d0-a812-48ade5032a2d None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:54:24.638414 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07eafc8e-8015-43d0-a812-48ade5032a2d None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:54:24.638537 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07eafc8e-8015-43d0-a812-48ade5032a2d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:54:24.712279 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-07eafc8e-8015-43d0-a812-48ade5032a2d None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:54:24.713460 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-07eafc8e-8015-43d0-a812-48ade5032a2d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:54:24.739268 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:54:24.807482 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 555/1095] 127.0.0.1 () {58 vars in 998 bytes} [Thu Oct 20 08:54:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 20 08:54:24.954863 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb85b304-3326-4d1d-aee3-2053b03ec204 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:54:24.956288 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-eb85b304-3326-4d1d-aee3-2053b03ec204 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:54:25.098720 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb85b304-3326-4d1d-aee3-2053b03ec204 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:54:25.138683 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:54:25.210675 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e0a81f-4096-41e2-8e79-16668e2ecc77 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a94e868115ec49918eda10724263930c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:54:25.211206 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e0a81f-4096-41e2-8e79-16668e2ecc77 service nova] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:54:25.211486 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e0a81f-4096-41e2-8e79-16668e2ecc77 service nova] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:54:25.211811 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e0a81f-4096-41e2-8e79-16668e2ecc77 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:54:25.213412 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-08e0a81f-4096-41e2-8e79-16668e2ecc77 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:54:25.302357 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08e0a81f-4096-41e2-8e79-16668e2ecc77 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:54:25.303983 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08e0a81f-4096-41e2-8e79-16668e2ecc77 service nova] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:54:25.321510 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 541/1096] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 08:54:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 367 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 08:54:26.443220 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c96409ae-76dc-467b-9d13-d91a17defa78 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:54:26.443738 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c96409ae-76dc-467b-9d13-d91a17defa78 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:54:26.443847 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c96409ae-76dc-467b-9d13-d91a17defa78 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:54:26.445222 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 556/1097] 127.0.0.1 () {54 vars in 901 bytes} [Thu Oct 20 08:54:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:54:26.451323 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b360ceb-3a8a-4116-b803-a13a1f9cf0ea None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:54:26.451709 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b360ceb-3a8a-4116-b803-a13a1f9cf0ea None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:54:26.451858 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b360ceb-3a8a-4116-b803-a13a1f9cf0ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:54:26.505101 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:54:26.526344 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-7b360ceb-3a8a-4116-b803-a13a1f9cf0ea None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:54:26.527552 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-7b360ceb-3a8a-4116-b803-a13a1f9cf0ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:54:26.625332 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 542/1098] 127.0.0.1 () {58 vars in 998 bytes} [Thu Oct 20 08:54:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:54:26.777512 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c96409ae-76dc-467b-9d13-d91a17defa78 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:54:26.779597 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-c96409ae-76dc-467b-9d13-d91a17defa78 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:54:26.922636 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c96409ae-76dc-467b-9d13-d91a17defa78 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:54:26.934484 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:54:27.034346 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49c0f79c-16df-4f5b-b418-cca32f4ff2ae service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a94e868115ec49918eda10724263930c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:54:27.034973 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c0f79c-16df-4f5b-b418-cca32f4ff2ae service nova] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:54:27.035248 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c0f79c-16df-4f5b-b418-cca32f4ff2ae service nova] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:54:27.035515 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c0f79c-16df-4f5b-b418-cca32f4ff2ae service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:54:27.037143 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-49c0f79c-16df-4f5b-b418-cca32f4ff2ae service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:54:27.126879 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49c0f79c-16df-4f5b-b418-cca32f4ff2ae service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:54:27.128472 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49c0f79c-16df-4f5b-b418-cca32f4ff2ae service nova] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:54:27.146416 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 557/1099] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 08:54:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 369 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 08:54:28.250017 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ebdc94-828f-4444-a65d-1b91693463d3 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:54:28.250575 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ebdc94-828f-4444-a65d-1b91693463d3 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:54:28.250693 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ebdc94-828f-4444-a65d-1b91693463d3 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:54:28.251973 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 543/1100] 127.0.0.1 () {54 vars in 901 bytes} [Thu Oct 20 08:54:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:54:28.258092 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b040ee-f69d-4f66-b889-05b3341b9fb9 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:54:28.258546 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b040ee-f69d-4f66-b889-05b3341b9fb9 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:54:28.258670 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b040ee-f69d-4f66-b889-05b3341b9fb9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:54:28.277755 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:54:28.333735 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-b5b040ee-f69d-4f66-b889-05b3341b9fb9 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:54:28.334964 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-b5b040ee-f69d-4f66-b889-05b3341b9fb9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:54:28.430787 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 558/1101] 127.0.0.1 () {58 vars in 998 bytes} [Thu Oct 20 08:54:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 20 08:54:28.581617 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6ebdc94-828f-4444-a65d-1b91693463d3 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:54:28.583108 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-e6ebdc94-828f-4444-a65d-1b91693463d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:54:28.691418 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:54:28.727630 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6ebdc94-828f-4444-a65d-1b91693463d3 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:54:28.837218 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0251299d-dc86-4da3-b134-fe72de988c38 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a94e868115ec49918eda10724263930c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:54:28.837742 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0251299d-dc86-4da3-b134-fe72de988c38 service nova] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:54:28.838017 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0251299d-dc86-4da3-b134-fe72de988c38 service nova] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:54:28.838284 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0251299d-dc86-4da3-b134-fe72de988c38 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:54:28.839886 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-0251299d-dc86-4da3-b134-fe72de988c38 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:54:28.917254 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:54:28.931744 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0251299d-dc86-4da3-b134-fe72de988c38 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:54:28.933344 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0251299d-dc86-4da3-b134-fe72de988c38 service nova] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:54:28.951105 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 544/1102] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 08:54:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 370 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 08:54:30.042141 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e1c379-0564-4288-8eaf-0d26271d9674 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:54:30.042727 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e1c379-0564-4288-8eaf-0d26271d9674 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:54:30.042727 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e1c379-0564-4288-8eaf-0d26271d9674 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:54:30.044592 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 559/1103] 127.0.0.1 () {54 vars in 901 bytes} [Thu Oct 20 08:54:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:54:30.050782 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e77c3d50-6af9-49c9-8f18-f07cd169acc5 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:54:30.051235 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e77c3d50-6af9-49c9-8f18-f07cd169acc5 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:54:30.051331 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e77c3d50-6af9-49c9-8f18-f07cd169acc5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:54:30.128429 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-e77c3d50-6af9-49c9-8f18-f07cd169acc5 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:54:30.129617 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-e77c3d50-6af9-49c9-8f18-f07cd169acc5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:54:30.224185 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 545/1104] 127.0.0.1 () {58 vars in 998 bytes} [Thu Oct 20 08:54:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:54:30.373133 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7e1c379-0564-4288-8eaf-0d26271d9674 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:54:30.374800 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-f7e1c379-0564-4288-8eaf-0d26271d9674 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:54:30.456948 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:54:30.518057 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7e1c379-0564-4288-8eaf-0d26271d9674 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:54:30.631700 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2552f65d-b119-40b7-b6b1-b855683bc9d1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a94e868115ec49918eda10724263930c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:54:30.632205 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2552f65d-b119-40b7-b6b1-b855683bc9d1 service nova] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:54:30.632487 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2552f65d-b119-40b7-b6b1-b855683bc9d1 service nova] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:54:30.632757 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2552f65d-b119-40b7-b6b1-b855683bc9d1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:54:30.634345 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-2552f65d-b119-40b7-b6b1-b855683bc9d1 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:54:30.684108 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:54:30.724971 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2552f65d-b119-40b7-b6b1-b855683bc9d1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:54:30.726588 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2552f65d-b119-40b7-b6b1-b855683bc9d1 service nova] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:54:30.744653 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 560/1105] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 08:54:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 372 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 08:54:31.783462 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8570a7df-d8b1-480c-8a52-24b1dd12678d None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:54:31.784047 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8570a7df-d8b1-480c-8a52-24b1dd12678d None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:54:31.784047 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8570a7df-d8b1-480c-8a52-24b1dd12678d None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:54:31.785243 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 546/1106] 127.0.0.1 () {54 vars in 901 bytes} [Thu Oct 20 08:54:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:54:31.791318 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84f43b0-d785-4c37-9c27-a1adbea317bc None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:54:31.791741 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84f43b0-d785-4c37-9c27-a1adbea317bc None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:54:31.791897 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84f43b0-d785-4c37-9c27-a1adbea317bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:54:31.867494 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-f84f43b0-d785-4c37-9c27-a1adbea317bc None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:54:31.868677 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-f84f43b0-d785-4c37-9c27-a1adbea317bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:54:31.963556 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 561/1107] 127.0.0.1 () {58 vars in 998 bytes} [Thu Oct 20 08:54:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:54:32.111264 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8570a7df-d8b1-480c-8a52-24b1dd12678d None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:54:32.112755 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-8570a7df-d8b1-480c-8a52-24b1dd12678d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:54:32.161265 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:54:32.257717 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8570a7df-d8b1-480c-8a52-24b1dd12678d None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:54:32.372463 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8685cf73-d029-450f-a5d9-8260d9cd3540 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a94e868115ec49918eda10724263930c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:54:32.372993 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8685cf73-d029-450f-a5d9-8260d9cd3540 service nova] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:54:32.373267 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8685cf73-d029-450f-a5d9-8260d9cd3540 service nova] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:54:32.373537 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8685cf73-d029-450f-a5d9-8260d9cd3540 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:54:32.375149 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-8685cf73-d029-450f-a5d9-8260d9cd3540 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:54:32.382496 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:54:32.468122 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8685cf73-d029-450f-a5d9-8260d9cd3540 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:54:32.469716 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8685cf73-d029-450f-a5d9-8260d9cd3540 service nova] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:54:32.487319 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 547/1108] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 08:54:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 377 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 08:54:33.532479 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b141f4-b5c3-499c-a919-e90297ae5bca None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:54:33.533038 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b141f4-b5c3-499c-a919-e90297ae5bca None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:54:33.533038 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b141f4-b5c3-499c-a919-e90297ae5bca None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:54:33.534376 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 562/1109] 127.0.0.1 () {54 vars in 901 bytes} [Thu Oct 20 08:54:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:54:33.540519 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f256a1e9-9d91-4845-9e35-b30f5b779b29 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:54:33.540947 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f256a1e9-9d91-4845-9e35-b30f5b779b29 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:54:33.541045 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f256a1e9-9d91-4845-9e35-b30f5b779b29 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:54:33.618015 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-f256a1e9-9d91-4845-9e35-b30f5b779b29 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:54:33.619240 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-f256a1e9-9d91-4845-9e35-b30f5b779b29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:54:33.685787 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:54:33.715655 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 548/1110] 127.0.0.1 () {58 vars in 998 bytes} [Thu Oct 20 08:54:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:54:33.865763 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b141f4-b5c3-499c-a919-e90297ae5bca None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:54:33.867305 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-93b141f4-b5c3-499c-a919-e90297ae5bca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:54:33.884440 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:54:34.010097 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b141f4-b5c3-499c-a919-e90297ae5bca None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:54:34.102666 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:54:34.129200 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2c4e25c-b220-444c-909c-ab48b618148e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a94e868115ec49918eda10724263930c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:54:34.129798 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c4e25c-b220-444c-909c-ab48b618148e service nova] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:54:34.130079 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c4e25c-b220-444c-909c-ab48b618148e service nova] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:54:34.130350 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c4e25c-b220-444c-909c-ab48b618148e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:54:34.131958 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-d2c4e25c-b220-444c-909c-ab48b618148e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:54:34.222636 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2c4e25c-b220-444c-909c-ab48b618148e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:54:34.224316 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2c4e25c-b220-444c-909c-ab48b618148e service nova] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:54:34.242493 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 563/1111] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 08:54:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 377 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 08:54:35.277232 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c15eac-882a-44cc-a328-83fff0590b20 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:54:35.278308 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c15eac-882a-44cc-a328-83fff0590b20 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:54:35.278308 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c15eac-882a-44cc-a328-83fff0590b20 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:54:35.279648 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 549/1112] 127.0.0.1 () {54 vars in 901 bytes} [Thu Oct 20 08:54:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:54:35.285725 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8950065-d480-4388-8f08-cf0c91339617 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:54:35.286129 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8950065-d480-4388-8f08-cf0c91339617 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:54:35.286269 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8950065-d480-4388-8f08-cf0c91339617 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:54:35.362553 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-f8950065-d480-4388-8f08-cf0c91339617 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:54:35.363747 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-f8950065-d480-4388-8f08-cf0c91339617 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:54:35.406134 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:54:35.459400 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 564/1113] 127.0.0.1 () {58 vars in 998 bytes} [Thu Oct 20 08:54:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:54:35.606696 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80c15eac-882a-44cc-a328-83fff0590b20 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:54:35.608227 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-80c15eac-882a-44cc-a328-83fff0590b20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:54:35.752743 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80c15eac-882a-44cc-a328-83fff0590b20 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:54:35.806298 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:54:35.865073 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53c471a9-d9c1-4812-8468-09d638891f76 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a94e868115ec49918eda10724263930c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:54:35.865603 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53c471a9-d9c1-4812-8468-09d638891f76 service nova] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:54:35.865877 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53c471a9-d9c1-4812-8468-09d638891f76 service nova] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:54:35.866143 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53c471a9-d9c1-4812-8468-09d638891f76 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:54:35.867763 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-53c471a9-d9c1-4812-8468-09d638891f76 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:54:35.956947 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53c471a9-d9c1-4812-8468-09d638891f76 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:54:35.958595 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53c471a9-d9c1-4812-8468-09d638891f76 service nova] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:54:35.976889 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 550/1114] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 08:54:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 08:54:37.034506 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e3692f-49b2-4030-bd48-2b54ea56e679 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:54:37.035137 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e3692f-49b2-4030-bd48-2b54ea56e679 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:54:37.035137 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e3692f-49b2-4030-bd48-2b54ea56e679 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:54:37.036327 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 565/1115] 127.0.0.1 () {54 vars in 901 bytes} [Thu Oct 20 08:54:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:54:37.042615 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78c71768-5d2c-4387-b70b-66adf4bc7f20 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:54:37.043006 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78c71768-5d2c-4387-b70b-66adf4bc7f20 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:54:37.043674 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78c71768-5d2c-4387-b70b-66adf4bc7f20 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:54:37.120881 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-78c71768-5d2c-4387-b70b-66adf4bc7f20 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:54:37.122724 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-78c71768-5d2c-4387-b70b-66adf4bc7f20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:54:37.124725 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:54:37.217490 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 551/1116] 127.0.0.1 () {58 vars in 998 bytes} [Thu Oct 20 08:54:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:54:37.366989 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e3692f-49b2-4030-bd48-2b54ea56e679 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:54:37.368620 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-50e3692f-49b2-4030-bd48-2b54ea56e679 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:54:37.513231 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e3692f-49b2-4030-bd48-2b54ea56e679 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:54:37.540434 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:54:37.627373 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-694399fa-7824-4c94-999f-e28e460dadad service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a94e868115ec49918eda10724263930c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:54:37.627881 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-694399fa-7824-4c94-999f-e28e460dadad service nova] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:54:37.628153 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-694399fa-7824-4c94-999f-e28e460dadad service nova] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:54:37.628415 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-694399fa-7824-4c94-999f-e28e460dadad service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:54:37.629984 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-694399fa-7824-4c94-999f-e28e460dadad service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:54:37.716052 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-694399fa-7824-4c94-999f-e28e460dadad service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:54:37.717601 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-694399fa-7824-4c94-999f-e28e460dadad service nova] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:54:37.735539 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 566/1117] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 08:54:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 369 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 08:54:39.787294 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8804c97c-e2bd-44b4-b493-ade8cdc7268c None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:54:39.787805 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8804c97c-e2bd-44b4-b493-ade8cdc7268c None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:54:39.787805 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8804c97c-e2bd-44b4-b493-ade8cdc7268c None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:54:39.789084 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 552/1118] 127.0.0.1 () {54 vars in 917 bytes} [Thu Oct 20 08:54:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:54:39.795020 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b1cc6b-3e61-4a69-92f5-98f1ccbb60ca None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:54:39.795465 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b1cc6b-3e61-4a69-92f5-98f1ccbb60ca None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:54:39.795569 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b1cc6b-3e61-4a69-92f5-98f1ccbb60ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:54:39.836729 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:54:39.862941 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-20b1cc6b-3e61-4a69-92f5-98f1ccbb60ca None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:54:39.864137 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-20b1cc6b-3e61-4a69-92f5-98f1ccbb60ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:54:39.959533 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 567/1119] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Oct 20 08:54:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:54:40.247294 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8804c97c-e2bd-44b4-b493-ade8cdc7268c None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:54:40.248810 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-8804c97c-e2bd-44b4-b493-ade8cdc7268c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:54:40.351005 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8804c97c-e2bd-44b4-b493-ade8cdc7268c None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:54:40.385700 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:54:40.426945 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3026364-3093-4a15-9cc5-82a2c8aa9419 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:54:40.427540 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3026364-3093-4a15-9cc5-82a2c8aa9419 service neutron] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:54:40.427812 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3026364-3093-4a15-9cc5-82a2c8aa9419 service neutron] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:54:40.428074 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3026364-3093-4a15-9cc5-82a2c8aa9419 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:54:40.429633 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-b3026364-3093-4a15-9cc5-82a2c8aa9419 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:54:40.536709 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3026364-3093-4a15-9cc5-82a2c8aa9419 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:54:40.538105 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3026364-3093-4a15-9cc5-82a2c8aa9419 service neutron] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:54:40.558166 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 553/1120] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 08:54:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 08:54:42.904125 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb3ff261-b435-40e2-a0f4-b04d930531ec None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:54:42.904678 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb3ff261-b435-40e2-a0f4-b04d930531ec None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:54:42.904678 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb3ff261-b435-40e2-a0f4-b04d930531ec None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:54:42.905997 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 568/1121] 127.0.0.1 () {54 vars in 917 bytes} [Thu Oct 20 08:54:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:54:42.911505 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-257293eb-7121-4541-8ab0-8737da140bc9 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:54:42.911956 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-257293eb-7121-4541-8ab0-8737da140bc9 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:54:42.912061 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-257293eb-7121-4541-8ab0-8737da140bc9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:54:42.977116 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-257293eb-7121-4541-8ab0-8737da140bc9 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:54:42.978296 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-257293eb-7121-4541-8ab0-8737da140bc9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:54:42.991515 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:54:43.075530 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 554/1122] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Oct 20 08:54:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:54:43.354766 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb3ff261-b435-40e2-a0f4-b04d930531ec None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:54:43.356989 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-bb3ff261-b435-40e2-a0f4-b04d930531ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:54:43.457973 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb3ff261-b435-40e2-a0f4-b04d930531ec None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:54:43.470755 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:54:43.530582 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14429c81-0329-4a98-bcfa-8f2b920c8f32 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:54:43.531177 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14429c81-0329-4a98-bcfa-8f2b920c8f32 service neutron] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:54:43.531449 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14429c81-0329-4a98-bcfa-8f2b920c8f32 service neutron] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:54:43.531715 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14429c81-0329-4a98-bcfa-8f2b920c8f32 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:54:43.533281 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-14429c81-0329-4a98-bcfa-8f2b920c8f32 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:54:43.641738 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14429c81-0329-4a98-bcfa-8f2b920c8f32 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:54:43.643220 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14429c81-0329-4a98-bcfa-8f2b920c8f32 service neutron] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:54:43.660545 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 569/1123] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 08:54:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 08:54:46.073048 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df7371a-e6c1-4832-b662-378f4960b06d None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:54:46.073584 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df7371a-e6c1-4832-b662-378f4960b06d None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:54:46.073584 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df7371a-e6c1-4832-b662-378f4960b06d None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:54:46.074893 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 555/1124] 127.0.0.1 () {54 vars in 917 bytes} [Thu Oct 20 08:54:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:54:46.080072 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-539fcdb3-0305-4e76-bcb6-621e4ee5904a None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:54:46.080498 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-539fcdb3-0305-4e76-bcb6-621e4ee5904a None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:54:46.080613 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-539fcdb3-0305-4e76-bcb6-621e4ee5904a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:54:46.129536 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:54:46.147383 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-539fcdb3-0305-4e76-bcb6-621e4ee5904a None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:54:46.148575 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-539fcdb3-0305-4e76-bcb6-621e4ee5904a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:54:46.242753 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 570/1125] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Oct 20 08:54:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:54:46.525535 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3df7371a-e6c1-4832-b662-378f4960b06d None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:54:46.527100 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-3df7371a-e6c1-4832-b662-378f4960b06d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:54:46.629760 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3df7371a-e6c1-4832-b662-378f4960b06d None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:54:46.670012 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:54:46.703367 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8578127-e63e-4db1-bfe2-91f58b5e6c9b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:54:46.703954 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8578127-e63e-4db1-bfe2-91f58b5e6c9b service neutron] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:54:46.704227 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8578127-e63e-4db1-bfe2-91f58b5e6c9b service neutron] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:54:46.704489 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8578127-e63e-4db1-bfe2-91f58b5e6c9b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:54:46.706050 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-f8578127-e63e-4db1-bfe2-91f58b5e6c9b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:54:46.813852 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8578127-e63e-4db1-bfe2-91f58b5e6c9b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:54:46.815322 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8578127-e63e-4db1-bfe2-91f58b5e6c9b service neutron] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:54:46.833329 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 556/1126] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 08:54:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 08:54:49.078013 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b959631e-5167-4da9-84ea-4b48d3541605 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:54:49.078590 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b959631e-5167-4da9-84ea-4b48d3541605 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:54:49.078590 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b959631e-5167-4da9-84ea-4b48d3541605 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:54:49.079935 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 571/1127] 127.0.0.1 () {54 vars in 917 bytes} [Thu Oct 20 08:54:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:54:49.085276 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f82a076-a39f-4b3c-947d-c11e788c2a8f None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:54:49.085980 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f82a076-a39f-4b3c-947d-c11e788c2a8f None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:54:49.086092 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f82a076-a39f-4b3c-947d-c11e788c2a8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:54:49.153403 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-6f82a076-a39f-4b3c-947d-c11e788c2a8f None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:54:49.154624 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-6f82a076-a39f-4b3c-947d-c11e788c2a8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:54:49.175377 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:54:49.251609 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 557/1128] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Oct 20 08:54:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:54:49.545304 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b959631e-5167-4da9-84ea-4b48d3541605 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:54:49.546875 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-b959631e-5167-4da9-84ea-4b48d3541605 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:54:49.648889 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b959631e-5167-4da9-84ea-4b48d3541605 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:54:49.669783 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:54:49.722762 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c774bad-00ba-4fe3-934d-7dbb761c9744 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:54:49.723353 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c774bad-00ba-4fe3-934d-7dbb761c9744 service neutron] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:54:49.723627 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c774bad-00ba-4fe3-934d-7dbb761c9744 service neutron] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:54:49.723893 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c774bad-00ba-4fe3-934d-7dbb761c9744 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:54:49.725452 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-6c774bad-00ba-4fe3-934d-7dbb761c9744 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:54:49.832672 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c774bad-00ba-4fe3-934d-7dbb761c9744 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:54:49.834122 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c774bad-00ba-4fe3-934d-7dbb761c9744 service neutron] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:54:49.852522 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 572/1129] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 08:54:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 08:54:52.034441 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb6a269-b759-42a5-b8c7-8c37f563e673 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:54:52.034999 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb6a269-b759-42a5-b8c7-8c37f563e673 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:54:52.034999 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb6a269-b759-42a5-b8c7-8c37f563e673 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:54:52.036347 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 558/1130] 127.0.0.1 () {54 vars in 917 bytes} [Thu Oct 20 08:54:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:54:52.041871 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c3d0150-e62d-4350-9301-3219a2e61b8b None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:54:52.042272 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c3d0150-e62d-4350-9301-3219a2e61b8b None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:54:52.042415 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c3d0150-e62d-4350-9301-3219a2e61b8b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:54:52.106061 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:54:52.109266 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-5c3d0150-e62d-4350-9301-3219a2e61b8b None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:54:52.111101 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-5c3d0150-e62d-4350-9301-3219a2e61b8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:54:52.204337 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 573/1131] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Oct 20 08:54:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:54:52.489446 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eb6a269-b759-42a5-b8c7-8c37f563e673 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:54:52.491018 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-3eb6a269-b759-42a5-b8c7-8c37f563e673 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:54:52.594147 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eb6a269-b759-42a5-b8c7-8c37f563e673 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:54:52.642422 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:54:52.666657 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b13cd53-def0-484f-b052-76e6be10b7c4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:54:52.667254 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b13cd53-def0-484f-b052-76e6be10b7c4 service neutron] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:54:52.667637 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b13cd53-def0-484f-b052-76e6be10b7c4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:54:52.667901 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b13cd53-def0-484f-b052-76e6be10b7c4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:54:52.669462 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-1b13cd53-def0-484f-b052-76e6be10b7c4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:54:52.774989 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b13cd53-def0-484f-b052-76e6be10b7c4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:54:52.776408 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b13cd53-def0-484f-b052-76e6be10b7c4 service neutron] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:54:52.794454 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 559/1132] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 08:54:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 08:54:54.976384 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca401ba-469c-439b-b8f6-89c46892a5f7 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:54:54.976957 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca401ba-469c-439b-b8f6-89c46892a5f7 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:54:54.976957 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca401ba-469c-439b-b8f6-89c46892a5f7 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:54:54.978273 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 574/1133] 127.0.0.1 () {54 vars in 917 bytes} [Thu Oct 20 08:54:54 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:54:54.984233 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1751e93e-3504-424a-930a-9df39380909f None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:54:54.984660 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1751e93e-3504-424a-930a-9df39380909f None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:54:54.984776 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1751e93e-3504-424a-930a-9df39380909f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:54:55.065222 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-1751e93e-3504-424a-930a-9df39380909f None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:54:55.066423 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-1751e93e-3504-424a-930a-9df39380909f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:54:55.101221 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:54:55.177588 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 560/1134] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Oct 20 08:54:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:54:55.461799 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ca401ba-469c-439b-b8f6-89c46892a5f7 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:54:55.463371 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-6ca401ba-469c-439b-b8f6-89c46892a5f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:54:55.566453 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ca401ba-469c-439b-b8f6-89c46892a5f7 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:54:55.595138 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:54:55.640668 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41515d97-e52f-4ded-a903-a5cc48d6b4c3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:54:55.641193 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41515d97-e52f-4ded-a903-a5cc48d6b4c3 service neutron] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:54:55.641466 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41515d97-e52f-4ded-a903-a5cc48d6b4c3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:54:55.641732 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41515d97-e52f-4ded-a903-a5cc48d6b4c3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:54:55.643367 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-41515d97-e52f-4ded-a903-a5cc48d6b4c3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:54:55.752127 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41515d97-e52f-4ded-a903-a5cc48d6b4c3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:54:55.753550 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41515d97-e52f-4ded-a903-a5cc48d6b4c3 service neutron] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:54:55.771719 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 575/1135] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 08:54:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 08:54:57.950678 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb7b66a-01ef-4dd4-8b93-d155c4e23c6f None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:54:57.951211 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb7b66a-01ef-4dd4-8b93-d155c4e23c6f None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:54:57.951211 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb7b66a-01ef-4dd4-8b93-d155c4e23c6f None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:54:57.952627 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 561/1136] 127.0.0.1 () {54 vars in 917 bytes} [Thu Oct 20 08:54:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 20 08:54:57.957967 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d540e1-7f69-4a75-bc7d-01e2d082f372 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:54:57.958420 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d540e1-7f69-4a75-bc7d-01e2d082f372 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:54:57.958541 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d540e1-7f69-4a75-bc7d-01e2d082f372 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:54:58.024023 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-27d540e1-7f69-4a75-bc7d-01e2d082f372 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:54:58.025206 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-27d540e1-7f69-4a75-bc7d-01e2d082f372 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:54:58.033331 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:54:58.121963 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 576/1137] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Oct 20 08:54:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 20 08:54:58.406537 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eb7b66a-01ef-4dd4-8b93-d155c4e23c6f None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:54:58.408063 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-4eb7b66a-01ef-4dd4-8b93-d155c4e23c6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:54:58.508756 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eb7b66a-01ef-4dd4-8b93-d155c4e23c6f None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:54:58.563194 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:54:58.585358 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4c5c2f1-224f-4833-9194-bfdf947225f4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:54:58.586607 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c5c2f1-224f-4833-9194-bfdf947225f4 service neutron] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:54:58.587062 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c5c2f1-224f-4833-9194-bfdf947225f4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:54:58.587507 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c5c2f1-224f-4833-9194-bfdf947225f4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:54:58.590908 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-e4c5c2f1-224f-4833-9194-bfdf947225f4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:54:58.696097 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4c5c2f1-224f-4833-9194-bfdf947225f4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:54:58.697452 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4c5c2f1-224f-4833-9194-bfdf947225f4 service neutron] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:54:58.716111 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 562/1138] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 08:54:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 08:55:00.885848 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e63210-7e0c-4659-a60c-6153cb660fb4 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:55:00.886396 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e63210-7e0c-4659-a60c-6153cb660fb4 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:55:00.887121 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e63210-7e0c-4659-a60c-6153cb660fb4 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:55:00.888315 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 577/1139] 127.0.0.1 () {54 vars in 917 bytes} [Thu Oct 20 08:55:00 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:55:00.893436 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f1974f-17ee-4780-ac03-f2280a83b07f None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:55:00.893843 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f1974f-17ee-4780-ac03-f2280a83b07f None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:55:00.893980 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f1974f-17ee-4780-ac03-f2280a83b07f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:55:00.960022 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-f6f1974f-17ee-4780-ac03-f2280a83b07f None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:55:00.961465 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-f6f1974f-17ee-4780-ac03-f2280a83b07f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:55:01.001906 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:55:01.061557 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 563/1140] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Oct 20 08:55:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:55:01.342916 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e63210-7e0c-4659-a60c-6153cb660fb4 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:55:01.344490 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-59e63210-7e0c-4659-a60c-6153cb660fb4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:55:01.446654 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e63210-7e0c-4659-a60c-6153cb660fb4 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:55:01.484107 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:55:01.520345 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9976589d-d9be-42f8-a16f-23ed376410d0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:55:01.520940 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9976589d-d9be-42f8-a16f-23ed376410d0 service neutron] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:55:01.521211 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9976589d-d9be-42f8-a16f-23ed376410d0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:55:01.521473 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9976589d-d9be-42f8-a16f-23ed376410d0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:55:01.523074 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-9976589d-d9be-42f8-a16f-23ed376410d0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:55:01.630442 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9976589d-d9be-42f8-a16f-23ed376410d0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:55:01.631905 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9976589d-d9be-42f8-a16f-23ed376410d0 service neutron] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:55:01.649654 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 578/1141] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 08:55:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 08:55:03.828897 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aca4b5a-270d-4d56-b86e-1ec332bdc957 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:55:03.829373 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aca4b5a-270d-4d56-b86e-1ec332bdc957 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:55:03.829478 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aca4b5a-270d-4d56-b86e-1ec332bdc957 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:55:03.830838 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 564/1142] 127.0.0.1 () {54 vars in 917 bytes} [Thu Oct 20 08:55:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:55:03.837974 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f1313f-0fcb-4155-a1cf-099d36425b48 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:55:03.838404 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f1313f-0fcb-4155-a1cf-099d36425b48 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:55:03.838593 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f1313f-0fcb-4155-a1cf-099d36425b48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:55:03.903817 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-b6f1313f-0fcb-4155-a1cf-099d36425b48 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:55:03.905009 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-b6f1313f-0fcb-4155-a1cf-099d36425b48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:55:03.922934 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:55:04.000955 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 579/1143] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Oct 20 08:55:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:55:04.281857 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aca4b5a-270d-4d56-b86e-1ec332bdc957 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:55:04.283439 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-6aca4b5a-270d-4d56-b86e-1ec332bdc957 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:55:04.390925 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aca4b5a-270d-4d56-b86e-1ec332bdc957 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:55:04.457282 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:55:04.465098 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7196d179-c723-4f7c-9958-dc813d5685c9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:55:04.465724 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7196d179-c723-4f7c-9958-dc813d5685c9 service neutron] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:55:04.465997 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7196d179-c723-4f7c-9958-dc813d5685c9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:55:04.466261 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7196d179-c723-4f7c-9958-dc813d5685c9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:55:04.467906 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-7196d179-c723-4f7c-9958-dc813d5685c9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:55:04.573457 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7196d179-c723-4f7c-9958-dc813d5685c9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:55:04.575446 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7196d179-c723-4f7c-9958-dc813d5685c9 service neutron] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:55:04.594073 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 565/1144] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 08:55:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 08:55:06.734835 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf1b6a7f-2dc6-4d1a-bb28-1e13e26d7106 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:55:06.735396 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf1b6a7f-2dc6-4d1a-bb28-1e13e26d7106 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:55:06.735396 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf1b6a7f-2dc6-4d1a-bb28-1e13e26d7106 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:55:06.736666 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 580/1145] 127.0.0.1 () {54 vars in 917 bytes} [Thu Oct 20 08:55:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:55:06.741967 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3836c50d-43db-4570-b0c0-6d8eb5ca555c None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:55:06.742415 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3836c50d-43db-4570-b0c0-6d8eb5ca555c None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:55:06.742545 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3836c50d-43db-4570-b0c0-6d8eb5ca555c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:55:06.807328 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-3836c50d-43db-4570-b0c0-6d8eb5ca555c None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:55:06.808554 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-3836c50d-43db-4570-b0c0-6d8eb5ca555c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:55:06.857691 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:55:06.905128 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 566/1146] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Oct 20 08:55:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:55:07.187267 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf1b6a7f-2dc6-4d1a-bb28-1e13e26d7106 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:55:07.188872 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-cf1b6a7f-2dc6-4d1a-bb28-1e13e26d7106 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:55:07.289595 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf1b6a7f-2dc6-4d1a-bb28-1e13e26d7106 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:55:07.337454 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:55:07.362605 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a4a9d57-9974-401f-b29d-f189e85c7545 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:55:07.363232 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a4a9d57-9974-401f-b29d-f189e85c7545 service neutron] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:55:07.363503 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a4a9d57-9974-401f-b29d-f189e85c7545 service neutron] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:55:07.363766 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a4a9d57-9974-401f-b29d-f189e85c7545 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:55:07.365342 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-4a4a9d57-9974-401f-b29d-f189e85c7545 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:55:07.473001 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a4a9d57-9974-401f-b29d-f189e85c7545 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:55:07.474354 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a4a9d57-9974-401f-b29d-f189e85c7545 service neutron] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:55:07.491750 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 581/1147] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 08:55:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 08:55:16.903726 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=22 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:55:17.489513 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=66 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:55:27.332160 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f27c9ae6-fad1-4717-867d-5629db9a0fe4 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:55:27.332756 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f27c9ae6-fad1-4717-867d-5629db9a0fe4 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:55:27.332756 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f27c9ae6-fad1-4717-867d-5629db9a0fe4 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:55:27.334041 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 567/1148] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:55:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:55:27.340047 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff951ce-88cf-4b8c-ad78-3fa1f29b4b59 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:55:27.340482 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff951ce-88cf-4b8c-ad78-3fa1f29b4b59 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:55:27.340613 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff951ce-88cf-4b8c-ad78-3fa1f29b4b59 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:55:27.408324 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-9ff951ce-88cf-4b8c-ad78-3fa1f29b4b59 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:55:27.409525 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-9ff951ce-88cf-4b8c-ad78-3fa1f29b4b59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:55:27.503971 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 582/1149] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:55:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:55:27.806018 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730cb2c6-f5b5-4bfc-aea8-4721fe43436a None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:55:27.806306 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730cb2c6-f5b5-4bfc-aea8-4721fe43436a None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:55:27.806617 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730cb2c6-f5b5-4bfc-aea8-4721fe43436a None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:55:27.807977 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 568/1150] 10.222.0.36 () {54 vars in 962 bytes} [Thu Oct 20 08:55:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:55:27.812971 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de33012e-6b49-408c-b321-303965ed3eac None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:55:27.813257 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de33012e-6b49-408c-b321-303965ed3eac None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:55:27.813524 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de33012e-6b49-408c-b321-303965ed3eac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:55:27.889893 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-de33012e-6b49-408c-b321-303965ed3eac None None] MFA Rules not processed for user `ce043914cd9042909f70ab0adb272d34`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:55:27.891119 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-de33012e-6b49-408c-b321-303965ed3eac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:55:27.911373 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:55:27.966662 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 583/1151] 10.222.0.36 () {58 vars in 1059 bytes} [Thu Oct 20 08:55:27 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:55:27.974315 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-730cb2c6-f5b5-4bfc-aea8-4721fe43436a None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:55:27.975835 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-730cb2c6-f5b5-4bfc-aea8-4721fe43436a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:55:28.077336 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-730cb2c6-f5b5-4bfc-aea8-4721fe43436a None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:55:28.148083 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf5a73a-33ef-4580-a75a-7025b841bfe2 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce043914cd9042909f70ab0adb272d34', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:55:28.148674 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf5a73a-33ef-4580-a75a-7025b841bfe2 service glance] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:55:28.148945 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf5a73a-33ef-4580-a75a-7025b841bfe2 service glance] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:55:28.149209 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf5a73a-33ef-4580-a75a-7025b841bfe2 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:55:28.150793 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-cdf5a73a-33ef-4580-a75a-7025b841bfe2 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:55:28.204983 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:55:28.258411 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdf5a73a-33ef-4580-a75a-7025b841bfe2 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:55:28.259864 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdf5a73a-33ef-4580-a75a-7025b841bfe2 service glance] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:55:28.278226 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 569/1152] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 08:55:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 08:55:30.504213 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d6f9d4-0e90-4ad3-8081-8c182b95cd36 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:55:30.504750 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d6f9d4-0e90-4ad3-8081-8c182b95cd36 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:55:30.504750 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d6f9d4-0e90-4ad3-8081-8c182b95cd36 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:55:30.506110 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 584/1153] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:55:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:55:30.512028 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f242a168-00f9-4684-a262-d35e9af49036 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:55:30.512406 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f242a168-00f9-4684-a262-d35e9af49036 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:55:30.512607 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f242a168-00f9-4684-a262-d35e9af49036 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:55:30.577665 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-f242a168-00f9-4684-a262-d35e9af49036 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:55:30.578873 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-f242a168-00f9-4684-a262-d35e9af49036 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:55:30.675145 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 570/1154] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:55:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:55:30.944293 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d6f9d4-0e90-4ad3-8081-8c182b95cd36 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:55:30.945816 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-25d6f9d4-0e90-4ad3-8081-8c182b95cd36 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:55:31.073259 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d6f9d4-0e90-4ad3-8081-8c182b95cd36 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:55:31.124275 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:55:31.168389 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e08ec01d-ae80-4b29-9bc2-aedaf019da7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:55:31.168993 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e08ec01d-ae80-4b29-9bc2-aedaf019da7a admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:55:31.169271 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e08ec01d-ae80-4b29-9bc2-aedaf019da7a admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:55:31.169541 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e08ec01d-ae80-4b29-9bc2-aedaf019da7a admin admin] PATH_INFO: `/v3/projects` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:55:31.170175 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e08ec01d-ae80-4b29-9bc2-aedaf019da7a admin admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:55:31.171302 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: WARNING py.warnings [None req-e08ec01d-ae80-4b29-9bc2-aedaf019da7a admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 20 08:55:31.171302 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: warnings.warn(msg) Oct 20 08:55:31.171302 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: Oct 20 08:55:31.172456 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e08ec01d-ae80-4b29-9bc2-aedaf019da7a admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:55:31.191739 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 585/1155] 10.222.0.36 () {56 vars in 1108 bytes} [Thu Oct 20 08:55:30 2022] GET /identity/v3/projects => generated 2553 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 08:55:33.361742 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3abdf39b-b053-4a01-88b7-6f3e026e6eda None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:55:33.362306 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3abdf39b-b053-4a01-88b7-6f3e026e6eda None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:55:33.362306 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3abdf39b-b053-4a01-88b7-6f3e026e6eda None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:55:33.363581 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 571/1156] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:55:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:55:33.368962 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-535ec9fa-b40e-43c8-b29f-cb433076f242 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:55:33.369441 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-535ec9fa-b40e-43c8-b29f-cb433076f242 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:55:33.369554 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-535ec9fa-b40e-43c8-b29f-cb433076f242 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:55:33.435121 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-535ec9fa-b40e-43c8-b29f-cb433076f242 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:55:33.436314 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-535ec9fa-b40e-43c8-b29f-cb433076f242 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:55:33.529834 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 586/1157] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:55:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:55:33.826929 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3abdf39b-b053-4a01-88b7-6f3e026e6eda None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:55:33.828436 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-3abdf39b-b053-4a01-88b7-6f3e026e6eda None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:55:33.838505 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:55:33.975527 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3abdf39b-b053-4a01-88b7-6f3e026e6eda None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:55:34.053840 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:55:34.088802 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bcd588d-9b2f-4ead-a8ce-f6aa3a25b788 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a94e868115ec49918eda10724263930c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:55:34.089437 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bcd588d-9b2f-4ead-a8ce-f6aa3a25b788 service nova] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:55:34.089712 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bcd588d-9b2f-4ead-a8ce-f6aa3a25b788 service nova] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:55:34.089982 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bcd588d-9b2f-4ead-a8ce-f6aa3a25b788 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:55:34.091604 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-2bcd588d-9b2f-4ead-a8ce-f6aa3a25b788 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:55:34.180212 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bcd588d-9b2f-4ead-a8ce-f6aa3a25b788 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:55:34.181797 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bcd588d-9b2f-4ead-a8ce-f6aa3a25b788 service nova] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:55:34.199451 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 572/1158] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 08:55:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 373 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 08:55:36.524204 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79ec731-098d-4e6f-9ea7-c140b6f97359 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:55:36.524771 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79ec731-098d-4e6f-9ea7-c140b6f97359 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:55:36.524771 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79ec731-098d-4e6f-9ea7-c140b6f97359 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:55:36.526398 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 587/1159] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:55:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:55:36.531938 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9680fc6a-cd3c-4bba-a112-ce7bdd962fb5 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:55:36.532397 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9680fc6a-cd3c-4bba-a112-ce7bdd962fb5 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:55:36.532510 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9680fc6a-cd3c-4bba-a112-ce7bdd962fb5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:55:36.597254 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-9680fc6a-cd3c-4bba-a112-ce7bdd962fb5 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:55:36.598431 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-9680fc6a-cd3c-4bba-a112-ce7bdd962fb5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:55:36.648057 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:55:36.697321 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 573/1160] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:55:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:55:36.991025 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a79ec731-098d-4e6f-9ea7-c140b6f97359 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:55:36.992549 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-a79ec731-098d-4e6f-9ea7-c140b6f97359 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:55:37.016302 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:55:37.094203 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a79ec731-098d-4e6f-9ea7-c140b6f97359 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:55:37.168916 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28fbea30-f74d-4153-baba-f1ff635a758d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:55:37.169512 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28fbea30-f74d-4153-baba-f1ff635a758d service neutron] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:55:37.169786 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28fbea30-f74d-4153-baba-f1ff635a758d service neutron] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:55:37.170052 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28fbea30-f74d-4153-baba-f1ff635a758d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:55:37.171673 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-28fbea30-f74d-4153-baba-f1ff635a758d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:55:37.246163 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:55:37.280580 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28fbea30-f74d-4153-baba-f1ff635a758d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:55:37.282006 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28fbea30-f74d-4153-baba-f1ff635a758d service neutron] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:55:37.300372 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 588/1161] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 08:55:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 08:55:39.511638 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4169535e-97bc-4160-957e-1f3b324b6ff6 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:55:39.512191 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4169535e-97bc-4160-957e-1f3b324b6ff6 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:55:39.512191 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4169535e-97bc-4160-957e-1f3b324b6ff6 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:55:39.513515 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 574/1162] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:55:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:55:39.518826 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdbdecfc-4c20-4451-9777-30b4bb328af9 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:55:39.519268 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdbdecfc-4c20-4451-9777-30b4bb328af9 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:55:39.519379 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdbdecfc-4c20-4451-9777-30b4bb328af9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:55:39.583757 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-cdbdecfc-4c20-4451-9777-30b4bb328af9 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:55:39.584935 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-cdbdecfc-4c20-4451-9777-30b4bb328af9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:55:39.680316 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 589/1163] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:55:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:55:39.970695 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4169535e-97bc-4160-957e-1f3b324b6ff6 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:55:39.972262 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-4169535e-97bc-4160-957e-1f3b324b6ff6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:55:40.074559 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4169535e-97bc-4160-957e-1f3b324b6ff6 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:55:40.145312 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:55:40.146925 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a4ae87c-0d59-4a5d-937f-0ef2549073ff service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:55:40.147592 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4ae87c-0d59-4a5d-937f-0ef2549073ff service neutron] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:55:40.148048 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4ae87c-0d59-4a5d-937f-0ef2549073ff service neutron] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:55:40.148547 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4ae87c-0d59-4a5d-937f-0ef2549073ff service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:55:40.150420 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-5a4ae87c-0d59-4a5d-937f-0ef2549073ff service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:55:40.257256 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a4ae87c-0d59-4a5d-937f-0ef2549073ff service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:55:40.258642 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a4ae87c-0d59-4a5d-937f-0ef2549073ff service neutron] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:55:40.276511 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 575/1164] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 08:55:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 08:55:42.545951 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a3831c-8668-4c30-aab2-5faffb49a2b9 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:55:42.546569 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a3831c-8668-4c30-aab2-5faffb49a2b9 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:55:42.546569 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a3831c-8668-4c30-aab2-5faffb49a2b9 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:55:42.547846 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 590/1165] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:55:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:55:42.552979 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d21b72-4c41-4cd4-8f24-0b946f666cd1 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:55:42.553377 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d21b72-4c41-4cd4-8f24-0b946f666cd1 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:55:42.553557 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d21b72-4c41-4cd4-8f24-0b946f666cd1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:55:42.618517 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-57d21b72-4c41-4cd4-8f24-0b946f666cd1 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:55:42.619729 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-57d21b72-4c41-4cd4-8f24-0b946f666cd1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:55:42.673432 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:55:42.715565 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 576/1166] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:55:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:55:43.006669 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a3831c-8668-4c30-aab2-5faffb49a2b9 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:55:43.008210 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-a9a3831c-8668-4c30-aab2-5faffb49a2b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:55:43.044264 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:55:43.113374 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a3831c-8668-4c30-aab2-5faffb49a2b9 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:55:43.186604 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e5c6c3-8d21-4923-af89-aebd6ec65359 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:55:43.187208 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e5c6c3-8d21-4923-af89-aebd6ec65359 service neutron] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:55:43.187484 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e5c6c3-8d21-4923-af89-aebd6ec65359 service neutron] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:55:43.187752 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e5c6c3-8d21-4923-af89-aebd6ec65359 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:55:43.189340 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-f1e5c6c3-8d21-4923-af89-aebd6ec65359 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:55:43.268701 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:55:43.298490 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1e5c6c3-8d21-4923-af89-aebd6ec65359 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:55:43.299935 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1e5c6c3-8d21-4923-af89-aebd6ec65359 service neutron] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:55:43.317871 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 591/1167] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 08:55:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 08:55:45.652918 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d478e73e-1fad-4295-bb43-d12d3b5d0fcf None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:55:45.653446 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d478e73e-1fad-4295-bb43-d12d3b5d0fcf None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:55:45.653446 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d478e73e-1fad-4295-bb43-d12d3b5d0fcf None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:55:45.654753 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 577/1168] 10.222.0.36 () {54 vars in 927 bytes} [Thu Oct 20 08:55:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:55:45.660404 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09329ac5-7464-46cb-97c7-be62b0d8c42a None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:55:45.660838 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09329ac5-7464-46cb-97c7-be62b0d8c42a None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:55:45.660942 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09329ac5-7464-46cb-97c7-be62b0d8c42a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:55:45.726718 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-09329ac5-7464-46cb-97c7-be62b0d8c42a None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:55:45.727938 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-09329ac5-7464-46cb-97c7-be62b0d8c42a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:55:45.822287 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 592/1169] 10.222.0.36 () {58 vars in 1024 bytes} [Thu Oct 20 08:55:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:55:46.118089 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d478e73e-1fad-4295-bb43-d12d3b5d0fcf None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:55:46.119841 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-d478e73e-1fad-4295-bb43-d12d3b5d0fcf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:55:46.221898 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d478e73e-1fad-4295-bb43-d12d3b5d0fcf None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:55:46.293149 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91e6a146-c962-4267-9416-67bdcd54c5fd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:55:46.293731 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e6a146-c962-4267-9416-67bdcd54c5fd service neutron] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:55:46.294002 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e6a146-c962-4267-9416-67bdcd54c5fd service neutron] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:55:46.294265 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e6a146-c962-4267-9416-67bdcd54c5fd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:55:46.295854 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-91e6a146-c962-4267-9416-67bdcd54c5fd service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:55:46.305209 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:55:46.403769 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91e6a146-c962-4267-9416-67bdcd54c5fd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:55:46.405211 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91e6a146-c962-4267-9416-67bdcd54c5fd service neutron] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:55:46.423560 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 578/1170] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 08:55:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 08:55:55.820867 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=82 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:55:56.421311 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=50 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:57:07.173311 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-378cf0bd-dc30-4453-98e6-626d3fd0c94b None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:57:07.173910 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-378cf0bd-dc30-4453-98e6-626d3fd0c94b None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:57:07.173910 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-378cf0bd-dc30-4453-98e6-626d3fd0c94b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:57:07.248198 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-378cf0bd-dc30-4453-98e6-626d3fd0c94b None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:57:07.249385 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-378cf0bd-dc30-4453-98e6-626d3fd0c94b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:57:07.343866 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 593/1171] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 08:57:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:57:07.351845 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b36ae68-d7d0-4595-8fdb-ba13cfdd8393 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:57:07.352326 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b36ae68-d7d0-4595-8fdb-ba13cfdd8393 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:57:07.352431 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b36ae68-d7d0-4595-8fdb-ba13cfdd8393 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:57:07.428047 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-3b36ae68-d7d0-4595-8fdb-ba13cfdd8393 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:57:07.429187 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-3b36ae68-d7d0-4595-8fdb-ba13cfdd8393 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:57:07.523247 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 579/1172] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 08:57:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:57:07.528368 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-378cf0bd-dc30-4453-98e6-626d3fd0c94b None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:57:07.529556 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-378cf0bd-dc30-4453-98e6-626d3fd0c94b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:57:07.540595 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-378cf0bd-dc30-4453-98e6-626d3fd0c94b None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:57:07.600154 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:57:07.633484 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da0111fb-3918-4351-b464-2f5564e5c22a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:57:07.634070 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da0111fb-3918-4351-b464-2f5564e5c22a admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:57:07.634339 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da0111fb-3918-4351-b464-2f5564e5c22a admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:57:07.634640 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da0111fb-3918-4351-b464-2f5564e5c22a admin admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:57:07.635269 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da0111fb-3918-4351-b464-2f5564e5c22a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 08:57:07.635559 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da0111fb-3918-4351-b464-2f5564e5c22a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:57:07.637091 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: WARNING py.warnings [None req-da0111fb-3918-4351-b464-2f5564e5c22a admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 20 08:57:07.637091 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: warnings.warn(msg) Oct 20 08:57:07.637091 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: Oct 20 08:57:07.637976 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da0111fb-3918-4351-b464-2f5564e5c22a admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:57:07.648717 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 594/1173] 10.222.0.36 () {58 vars in 1153 bytes} [Thu Oct 20 08:57:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 121 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 08:57:07.658904 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b36ae68-d7d0-4595-8fdb-ba13cfdd8393 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:57:07.668093 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b36ae68-d7d0-4595-8fdb-ba13cfdd8393 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:57:07.728716 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:57:07.760803 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adeff852-66a3-45f3-a191-f6c1b5472cf3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:57:07.761907 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adeff852-66a3-45f3-a191-f6c1b5472cf3 admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:57:07.762181 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adeff852-66a3-45f3-a191-f6c1b5472cf3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:57:07.762444 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adeff852-66a3-45f3-a191-f6c1b5472cf3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:57:07.763098 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adeff852-66a3-45f3-a191-f6c1b5472cf3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:57:07.764102 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: WARNING py.warnings [None req-adeff852-66a3-45f3-a191-f6c1b5472cf3 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 20 08:57:07.764102 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: warnings.warn(msg) Oct 20 08:57:07.764102 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: Oct 20 08:57:07.765286 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adeff852-66a3-45f3-a191-f6c1b5472cf3 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:57:07.819335 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 580/1174] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 08:57:07 2022] POST /identity/v3/projects => generated 377 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 08:57:07.823936 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da0111fb-3918-4351-b464-2f5564e5c22a admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:57:07.825503 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-da0111fb-3918-4351-b464-2f5564e5c22a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:57:07.859906 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da0111fb-3918-4351-b464-2f5564e5c22a admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:57:07.951425 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-467b7bcf-edf5-4ae9-999f-5c0d8b660a3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:57:07.952021 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-467b7bcf-edf5-4ae9-999f-5c0d8b660a3b admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:57:07.952292 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-467b7bcf-edf5-4ae9-999f-5c0d8b660a3b admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:57:07.952555 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-467b7bcf-edf5-4ae9-999f-5c0d8b660a3b admin admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:57:07.953186 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-467b7bcf-edf5-4ae9-999f-5c0d8b660a3b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:57:07.954170 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: WARNING py.warnings [None req-467b7bcf-edf5-4ae9-999f-5c0d8b660a3b admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 20 08:57:07.954170 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: warnings.warn(msg) Oct 20 08:57:07.954170 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: Oct 20 08:57:07.955411 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-467b7bcf-edf5-4ae9-999f-5c0d8b660a3b admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:57:07.997009 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 595/1175] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 08:57:07 2022] POST /identity/v3/users => generated 289 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 08:57:08.001027 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adeff852-66a3-45f3-a191-f6c1b5472cf3 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:57:08.002563 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-adeff852-66a3-45f3-a191-f6c1b5472cf3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:57:08.063995 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adeff852-66a3-45f3-a191-f6c1b5472cf3 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:57:08.147095 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:57:08.158781 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d343fd6c-fc34-4097-82a8-709e33819d96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:57:08.159387 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d343fd6c-fc34-4097-82a8-709e33819d96 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:57:08.159665 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d343fd6c-fc34-4097-82a8-709e33819d96 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:57:08.159934 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d343fd6c-fc34-4097-82a8-709e33819d96 admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:57:08.160570 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d343fd6c-fc34-4097-82a8-709e33819d96 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:57:08.162166 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d343fd6c-fc34-4097-82a8-709e33819d96 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:57:08.174925 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 581/1176] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 08:57:08 2022] GET /identity/v3/roles => generated 1420 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 08:57:08.178854 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-467b7bcf-edf5-4ae9-999f-5c0d8b660a3b admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:57:08.180378 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-467b7bcf-edf5-4ae9-999f-5c0d8b660a3b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:57:08.191567 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-467b7bcf-edf5-4ae9-999f-5c0d8b660a3b admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:57:08.216604 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:57:08.286507 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d398aaa3-1095-4e3e-9aa5-b3a1dad84c7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:57:08.287042 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d398aaa3-1095-4e3e-9aa5-b3a1dad84c7d admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:57:08.287316 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d398aaa3-1095-4e3e-9aa5-b3a1dad84c7d admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:57:08.287580 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d398aaa3-1095-4e3e-9aa5-b3a1dad84c7d admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:57:08.288214 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d398aaa3-1095-4e3e-9aa5-b3a1dad84c7d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:57:08.289792 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d398aaa3-1095-4e3e-9aa5-b3a1dad84c7d admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:57:08.301799 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 596/1177] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 08:57:08 2022] GET /identity/v3/roles => generated 1420 bytes in 123 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 08:57:08.305871 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d343fd6c-fc34-4097-82a8-709e33819d96 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:57:08.307399 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-d343fd6c-fc34-4097-82a8-709e33819d96 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:57:08.340944 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d343fd6c-fc34-4097-82a8-709e33819d96 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:57:08.432101 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-473148aa-641e-4071-bafd-e283651b49ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:57:08.432737 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473148aa-641e-4071-bafd-e283651b49ec admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:57:08.433040 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473148aa-641e-4071-bafd-e283651b49ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:57:08.433328 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473148aa-641e-4071-bafd-e283651b49ec admin admin] PATH_INFO: `/v3/projects/01f29e13d82a488186754215ace10924/users/0dc5ec72108a4fe8bb5cb6e5989b0104/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:57:08.455609 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-473148aa-641e-4071-bafd-e283651b49ec admin admin] RBAC: Authorizing `identity:create_grant(project_id=01f29e13d82a488186754215ace10924, user_id=0dc5ec72108a4fe8bb5cb6e5989b0104, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:57:08.458266 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-473148aa-641e-4071-bafd-e283651b49ec admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:57:08.484538 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 582/1178] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 08:57:08 2022] PUT /identity/v3/projects/01f29e13d82a488186754215ace10924/users/0dc5ec72108a4fe8bb5cb6e5989b0104/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 08:57:08.495446 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95a7804-b40d-4659-91ba-067300d66be5 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:57:08.495816 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95a7804-b40d-4659-91ba-067300d66be5 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:57:08.496091 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95a7804-b40d-4659-91ba-067300d66be5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:57:08.552423 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-c95a7804-b40d-4659-91ba-067300d66be5 None None] MFA Rules not processed for user `0dc5ec72108a4fe8bb5cb6e5989b0104`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:57:08.553553 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-c95a7804-b40d-4659-91ba-067300d66be5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:57:08.629240 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 597/1179] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 08:57:08 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:57:08.642943 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f560bf-1fbd-45cf-9291-ffe2be163f10 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:57:08.643225 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f560bf-1fbd-45cf-9291-ffe2be163f10 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:57:08.643493 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f560bf-1fbd-45cf-9291-ffe2be163f10 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:57:08.644806 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 583/1180] 10.222.0.36 () {54 vars in 963 bytes} [Thu Oct 20 08:57:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:57:08.649790 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30df8677-f61f-4b7c-b6dc-48ecd4e29720 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:57:08.650080 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30df8677-f61f-4b7c-b6dc-48ecd4e29720 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:57:08.650352 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30df8677-f61f-4b7c-b6dc-48ecd4e29720 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:57:08.680749 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:57:08.725470 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-30df8677-f61f-4b7c-b6dc-48ecd4e29720 None None] MFA Rules not processed for user `7cfa6bb090de44eaa5bdf099a3096e8c`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:57:08.726681 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-30df8677-f61f-4b7c-b6dc-48ecd4e29720 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:57:08.799553 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 598/1181] 10.222.0.36 () {58 vars in 1060 bytes} [Thu Oct 20 08:57:08 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:57:08.808694 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45f560bf-1fbd-45cf-9291-ffe2be163f10 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:57:08.810160 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-45f560bf-1fbd-45cf-9291-ffe2be163f10 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:57:08.859898 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:57:08.913955 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45f560bf-1fbd-45cf-9291-ffe2be163f10 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:57:08.986608 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c7e599c-2d53-47a3-992d-810113c8bbb7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:57:08.987199 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c7e599c-2d53-47a3-992d-810113c8bbb7 service cinder] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:57:08.987468 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c7e599c-2d53-47a3-992d-810113c8bbb7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:57:08.987728 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c7e599c-2d53-47a3-992d-810113c8bbb7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:57:08.989283 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-1c7e599c-2d53-47a3-992d-810113c8bbb7 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:57:09.085267 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c7e599c-2d53-47a3-992d-810113c8bbb7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:57:09.086668 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c7e599c-2d53-47a3-992d-810113c8bbb7 service cinder] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:57:09.090367 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:57:09.105482 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 584/1182] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 08:57:08 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 298 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 08:57:09.150603 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de53eda-0122-44f0-93f4-26245782b515 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:57:09.150889 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de53eda-0122-44f0-93f4-26245782b515 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:57:09.151161 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de53eda-0122-44f0-93f4-26245782b515 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:57:09.152535 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 599/1183] 10.222.0.36 () {52 vars in 830 bytes} [Thu Oct 20 08:57:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:57:09.156059 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c7e599c-2d53-47a3-992d-810113c8bbb7 service cinder] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:57:09.157258 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-1c7e599c-2d53-47a3-992d-810113c8bbb7 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:57:09.256839 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c7e599c-2d53-47a3-992d-810113c8bbb7 service cinder] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:57:09.348918 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd0ec922-e39a-4bf8-840e-88db81381ba1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:57:09.349465 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd0ec922-e39a-4bf8-840e-88db81381ba1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:57:09.349744 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd0ec922-e39a-4bf8-840e-88db81381ba1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:57:09.350008 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd0ec922-e39a-4bf8-840e-88db81381ba1 admin admin] PATH_INFO: `/v3/users/0dc5ec72108a4fe8bb5cb6e5989b0104` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:57:09.363265 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd0ec922-e39a-4bf8-840e-88db81381ba1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0dc5ec72108a4fe8bb5cb6e5989b0104)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:57:09.364290 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: WARNING py.warnings [None req-cd0ec922-e39a-4bf8-840e-88db81381ba1 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 20 08:57:09.364290 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: warnings.warn(msg) Oct 20 08:57:09.364290 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: Oct 20 08:57:09.365521 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd0ec922-e39a-4bf8-840e-88db81381ba1 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:57:09.375929 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:57:09.430696 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-cd0ec922-e39a-4bf8-840e-88db81381ba1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0dc5ec72108a4fe8bb5cb6e5989b0104'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 08:57:09.439517 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-cd0ec922-e39a-4bf8-840e-88db81381ba1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0dc5ec72108a4fe8bb5cb6e5989b0104'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 08:57:09.462915 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-cd0ec922-e39a-4bf8-840e-88db81381ba1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0dc5ec72108a4fe8bb5cb6e5989b0104'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 08:57:09.476585 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-cd0ec922-e39a-4bf8-840e-88db81381ba1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0dc5ec72108a4fe8bb5cb6e5989b0104'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 08:57:09.477128 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-cd0ec922-e39a-4bf8-840e-88db81381ba1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0dc5ec72108a4fe8bb5cb6e5989b0104'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 08:57:09.482909 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 585/1184] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 08:57:09 2022] DELETE /identity/v3/users/0dc5ec72108a4fe8bb5cb6e5989b0104 => generated 0 bytes in 327 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 08:57:09.487282 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6161d072-5f6b-4f38-8b17-f16d1153630c None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:57:09.487753 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6161d072-5f6b-4f38-8b17-f16d1153630c None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:57:09.487861 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6161d072-5f6b-4f38-8b17-f16d1153630c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:57:09.544718 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-6161d072-5f6b-4f38-8b17-f16d1153630c None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:57:09.545894 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-6161d072-5f6b-4f38-8b17-f16d1153630c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:57:09.618698 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:57:09.631407 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 600/1185] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 08:57:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:57:09.641739 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd0ec922-e39a-4bf8-840e-88db81381ba1 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:57:09.642975 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-cd0ec922-e39a-4bf8-840e-88db81381ba1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:57:09.742447 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd0ec922-e39a-4bf8-840e-88db81381ba1 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:57:09.797319 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:57:09.816709 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ac2c90d-bcf0-4e2f-87f2-4128a893e6b2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:57:09.817313 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ac2c90d-bcf0-4e2f-87f2-4128a893e6b2 service neutron] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:57:09.817586 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ac2c90d-bcf0-4e2f-87f2-4128a893e6b2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:57:09.817853 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ac2c90d-bcf0-4e2f-87f2-4128a893e6b2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:57:09.819469 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-1ac2c90d-bcf0-4e2f-87f2-4128a893e6b2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:57:09.928152 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ac2c90d-bcf0-4e2f-87f2-4128a893e6b2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:57:09.929649 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ac2c90d-bcf0-4e2f-87f2-4128a893e6b2 service neutron] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:57:09.950124 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 586/1186] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 08:57:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 08:57:10.263760 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6161d072-5f6b-4f38-8b17-f16d1153630c None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:57:10.265118 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-6161d072-5f6b-4f38-8b17-f16d1153630c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:57:10.276560 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6161d072-5f6b-4f38-8b17-f16d1153630c None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:57:10.379944 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb522ed4-fb79-4487-965b-e9a66280050b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:57:10.380702 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb522ed4-fb79-4487-965b-e9a66280050b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:57:10.380833 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb522ed4-fb79-4487-965b-e9a66280050b admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:57:10.381096 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb522ed4-fb79-4487-965b-e9a66280050b admin admin] PATH_INFO: `/v3/projects/01f29e13d82a488186754215ace10924` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:57:10.391664 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb522ed4-fb79-4487-965b-e9a66280050b admin admin] RBAC: Authorizing `identity:delete_project(project_id=01f29e13d82a488186754215ace10924)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:57:10.392675 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: WARNING py.warnings [None req-fb522ed4-fb79-4487-965b-e9a66280050b admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 20 08:57:10.392675 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: warnings.warn(msg) Oct 20 08:57:10.392675 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: Oct 20 08:57:10.393883 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb522ed4-fb79-4487-965b-e9a66280050b admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:57:10.450799 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-fb522ed4-fb79-4487-965b-e9a66280050b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '01f29e13d82a488186754215ace10924', 'request_id': 'req-fb522ed4-fb79-4487-965b-e9a66280050b'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 08:57:10.464517 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-fb522ed4-fb79-4487-965b-e9a66280050b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '01f29e13d82a488186754215ace10924', 'request_id': 'req-fb522ed4-fb79-4487-965b-e9a66280050b'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 08:57:10.470636 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-fb522ed4-fb79-4487-965b-e9a66280050b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '01f29e13d82a488186754215ace10924', 'request_id': 'req-fb522ed4-fb79-4487-965b-e9a66280050b'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 08:57:10.498372 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-fb522ed4-fb79-4487-965b-e9a66280050b admin admin] The token cache is being invalidate because project 01f29e13d82a488186754215ace10924 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81562) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 08:57:10.499054 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-fb522ed4-fb79-4487-965b-e9a66280050b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 08:57:10.499172 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-fb522ed4-fb79-4487-965b-e9a66280050b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 08:57:10.500430 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 601/1187] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 08:57:10 2022] DELETE /identity/v3/projects/01f29e13d82a488186754215ace10924 => generated 0 bytes in 237 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 08:57:19.948054 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=62 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:57:20.495775 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=84,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:58:32.976518 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ace48472-b2bd-47ea-b654-477a36d1d451 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:58:32.977132 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ace48472-b2bd-47ea-b654-477a36d1d451 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:58:32.977132 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ace48472-b2bd-47ea-b654-477a36d1d451 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:58:32.978394 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 587/1188] 127.0.0.1 () {54 vars in 901 bytes} [Thu Oct 20 08:58:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:58:32.983688 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8dc6649-47c5-402f-b46d-e7f516a6fd72 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:58:32.984094 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8dc6649-47c5-402f-b46d-e7f516a6fd72 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:58:32.984243 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8dc6649-47c5-402f-b46d-e7f516a6fd72 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:58:33.065927 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-e8dc6649-47c5-402f-b46d-e7f516a6fd72 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:58:33.067217 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-e8dc6649-47c5-402f-b46d-e7f516a6fd72 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:58:33.168170 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 602/1189] 127.0.0.1 () {58 vars in 998 bytes} [Thu Oct 20 08:58:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:58:33.322617 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ace48472-b2bd-47ea-b654-477a36d1d451 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:58:33.324794 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-ace48472-b2bd-47ea-b654-477a36d1d451 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:58:33.443077 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ace48472-b2bd-47ea-b654-477a36d1d451 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:58:33.448446 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3c89116-19c0-4851-ba5d-f7b6f3fe00f1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a94e868115ec49918eda10724263930c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:58:33.449035 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c89116-19c0-4851-ba5d-f7b6f3fe00f1 service nova] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:58:33.449309 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c89116-19c0-4851-ba5d-f7b6f3fe00f1 service nova] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:58:33.449574 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c89116-19c0-4851-ba5d-f7b6f3fe00f1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:58:33.457348 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3c89116-19c0-4851-ba5d-f7b6f3fe00f1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:58:33.458968 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3c89116-19c0-4851-ba5d-f7b6f3fe00f1 service nova] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:58:33.468806 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 588/1190] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 08:58:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 147 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 08:58:34.541712 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-839e98fa-b42a-41ce-b879-21b45e78ae46 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:58:34.542246 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-839e98fa-b42a-41ce-b879-21b45e78ae46 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:58:34.542246 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-839e98fa-b42a-41ce-b879-21b45e78ae46 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:58:34.543583 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 603/1191] 127.0.0.1 () {54 vars in 901 bytes} [Thu Oct 20 08:58:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:58:34.550047 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d51212b-085d-43fc-9953-131c03610991 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:58:34.550556 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d51212b-085d-43fc-9953-131c03610991 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:58:34.550674 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d51212b-085d-43fc-9953-131c03610991 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:58:34.564144 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c7de681-543a-4efd-99e7-70d2b2a4add8 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:58:34.564423 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c7de681-543a-4efd-99e7-70d2b2a4add8 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:58:34.564690 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c7de681-543a-4efd-99e7-70d2b2a4add8 None None] PATH_INFO: `/v3` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:58:34.565935 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 604/1192] 10.222.0.23 () {54 vars in 925 bytes} [Thu Oct 20 08:58:34 2022] GET /identity/v3 => generated 255 bytes in 3 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 08:58:34.570280 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-4d51212b-085d-43fc-9953-131c03610991 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:58:34.571569 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-4d51212b-085d-43fc-9953-131c03610991 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:58:34.571752 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5dc6d52-6f8e-4a93-ac45-787bfe5d8dc6 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:58:34.572190 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5dc6d52-6f8e-4a93-ac45-787bfe5d8dc6 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:58:34.572463 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5dc6d52-6f8e-4a93-ac45-787bfe5d8dc6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:58:34.579749 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 589/1193] 127.0.0.1 () {58 vars in 998 bytes} [Thu Oct 20 08:58:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:58:34.632375 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-b5dc6d52-6f8e-4a93-ac45-787bfe5d8dc6 None None] MFA Rules not processed for user `ad03ab76b9c44b1cb369475afa25eddd`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:58:34.633550 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-b5dc6d52-6f8e-4a93-ac45-787bfe5d8dc6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:58:34.663581 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:58:34.705579 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 605/1194] 10.222.0.23 () {58 vars in 1015 bytes} [Thu Oct 20 08:58:34 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 20 08:58:34.731298 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d51212b-085d-43fc-9953-131c03610991 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:58:34.743298 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d51212b-085d-43fc-9953-131c03610991 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:58:34.748742 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20bf419b-45c1-43c6-b671-fe50809f1e08 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a94e868115ec49918eda10724263930c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:58:34.749335 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20bf419b-45c1-43c6-b671-fe50809f1e08 service nova] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:58:34.749610 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20bf419b-45c1-43c6-b671-fe50809f1e08 service nova] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:58:34.749880 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20bf419b-45c1-43c6-b671-fe50809f1e08 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:58:34.757866 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20bf419b-45c1-43c6-b671-fe50809f1e08 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:58:34.759474 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20bf419b-45c1-43c6-b671-fe50809f1e08 service nova] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:58:34.769114 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 590/1195] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 08:58:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 08:58:44.702710 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=17 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:58:44.765026 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=62 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:58:45.137804 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427b2cf1-189e-43cc-958d-8c81f5868b43 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:58:45.138092 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427b2cf1-189e-43cc-958d-8c81f5868b43 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:58:45.138362 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427b2cf1-189e-43cc-958d-8c81f5868b43 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:58:45.139807 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 606/1196] 10.222.0.36 () {54 vars in 961 bytes} [Thu Oct 20 08:58:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:58:45.144673 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-426120d0-518b-4ce7-9db6-bfda55a230b4 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:58:45.144957 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-426120d0-518b-4ce7-9db6-bfda55a230b4 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:58:45.145231 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-426120d0-518b-4ce7-9db6-bfda55a230b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:58:45.164371 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-426120d0-518b-4ce7-9db6-bfda55a230b4 None None] MFA Rules not processed for user `ad03ab76b9c44b1cb369475afa25eddd`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:58:45.165557 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-426120d0-518b-4ce7-9db6-bfda55a230b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:58:45.174384 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 591/1197] 10.222.0.36 () {58 vars in 1058 bytes} [Thu Oct 20 08:58:45 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:58:45.182191 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-427b2cf1-189e-43cc-958d-8c81f5868b43 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:58:45.196099 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-427b2cf1-189e-43cc-958d-8c81f5868b43 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:58:45.201119 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d59a113e-846c-4e57-af8a-545ad70b393b service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ad03ab76b9c44b1cb369475afa25eddd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:58:45.201711 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d59a113e-846c-4e57-af8a-545ad70b393b service placement] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:58:45.201984 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d59a113e-846c-4e57-af8a-545ad70b393b service placement] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:58:45.202260 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d59a113e-846c-4e57-af8a-545ad70b393b service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:58:45.210844 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d59a113e-846c-4e57-af8a-545ad70b393b service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:58:45.212441 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d59a113e-846c-4e57-af8a-545ad70b393b service placement] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:58:45.222966 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 607/1198] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 08:58:45 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 08:58:46.246283 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b2e8f5c-342e-41b4-aa34-894a09723e4f None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:58:46.246833 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b2e8f5c-342e-41b4-aa34-894a09723e4f None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:58:46.246833 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b2e8f5c-342e-41b4-aa34-894a09723e4f None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:58:46.248119 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 592/1199] 127.0.0.1 () {54 vars in 901 bytes} [Thu Oct 20 08:58:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:58:46.254287 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753b3eba-6535-4315-b666-fa19fe162c1e None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:58:46.254763 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753b3eba-6535-4315-b666-fa19fe162c1e None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:58:46.254870 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753b3eba-6535-4315-b666-fa19fe162c1e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:58:46.274831 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-753b3eba-6535-4315-b666-fa19fe162c1e None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:58:46.276021 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-753b3eba-6535-4315-b666-fa19fe162c1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:58:46.283613 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 608/1200] 127.0.0.1 () {58 vars in 998 bytes} [Thu Oct 20 08:58:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:58:46.437212 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b2e8f5c-342e-41b4-aa34-894a09723e4f None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:58:46.451790 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b2e8f5c-342e-41b4-aa34-894a09723e4f None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:58:46.457184 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3ad9a2-c6e4-4990-b594-d11f10d3db20 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a94e868115ec49918eda10724263930c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:58:46.457771 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3ad9a2-c6e4-4990-b594-d11f10d3db20 service nova] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:58:46.458053 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3ad9a2-c6e4-4990-b594-d11f10d3db20 service nova] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:58:46.458324 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3ad9a2-c6e4-4990-b594-d11f10d3db20 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:58:46.467155 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff3ad9a2-c6e4-4990-b594-d11f10d3db20 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:58:46.468787 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff3ad9a2-c6e4-4990-b594-d11f10d3db20 service nova] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:58:46.477954 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 593/1201] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 08:58:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 20 08:58:47.541541 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa4b551-5445-4589-bc0f-37ac1c41d2c9 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:58:47.542123 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa4b551-5445-4589-bc0f-37ac1c41d2c9 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:58:47.542123 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa4b551-5445-4589-bc0f-37ac1c41d2c9 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:58:47.543473 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 609/1202] 127.0.0.1 () {54 vars in 901 bytes} [Thu Oct 20 08:58:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:58:47.549415 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822d8e50-0f56-4a8d-84b0-7fda3ca83480 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:58:47.549813 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822d8e50-0f56-4a8d-84b0-7fda3ca83480 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:58:47.549957 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822d8e50-0f56-4a8d-84b0-7fda3ca83480 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:58:47.569364 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-822d8e50-0f56-4a8d-84b0-7fda3ca83480 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:58:47.570569 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-822d8e50-0f56-4a8d-84b0-7fda3ca83480 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:58:47.578919 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 594/1203] 127.0.0.1 () {58 vars in 998 bytes} [Thu Oct 20 08:58:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:58:47.732418 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fa4b551-5445-4589-bc0f-37ac1c41d2c9 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:58:47.747028 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fa4b551-5445-4589-bc0f-37ac1c41d2c9 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:58:47.752399 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b70b1f4-15fe-4880-ad3e-d0e4639c7007 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a94e868115ec49918eda10724263930c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:58:47.752981 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b70b1f4-15fe-4880-ad3e-d0e4639c7007 service nova] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:58:47.753279 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b70b1f4-15fe-4880-ad3e-d0e4639c7007 service nova] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:58:47.753551 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b70b1f4-15fe-4880-ad3e-d0e4639c7007 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:58:47.762067 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b70b1f4-15fe-4880-ad3e-d0e4639c7007 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:58:47.763682 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b70b1f4-15fe-4880-ad3e-d0e4639c7007 service nova] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:58:47.772813 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 610/1204] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 08:58:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 08:58:51.681681 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f471a5-1471-4d18-b4e0-e520a286c517 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:58:51.682208 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f471a5-1471-4d18-b4e0-e520a286c517 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:58:51.682208 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f471a5-1471-4d18-b4e0-e520a286c517 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:58:51.684042 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 595/1205] 127.0.0.1 () {54 vars in 917 bytes} [Thu Oct 20 08:58:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:58:51.689389 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c89a1908-66e3-463a-84a0-c710ce57aa81 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:58:51.689811 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c89a1908-66e3-463a-84a0-c710ce57aa81 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:58:51.689940 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c89a1908-66e3-463a-84a0-c710ce57aa81 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:58:51.709122 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-c89a1908-66e3-463a-84a0-c710ce57aa81 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:58:51.710313 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-c89a1908-66e3-463a-84a0-c710ce57aa81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:58:51.718308 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 611/1206] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Oct 20 08:58:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:58:52.007548 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6f471a5-1471-4d18-b4e0-e520a286c517 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:58:52.009272 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-c6f471a5-1471-4d18-b4e0-e520a286c517 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:58:52.085987 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6f471a5-1471-4d18-b4e0-e520a286c517 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:58:52.090339 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b80b94b-70ed-4ebb-b300-39d6f65076af service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:58:52.090958 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b80b94b-70ed-4ebb-b300-39d6f65076af service neutron] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:58:52.091237 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b80b94b-70ed-4ebb-b300-39d6f65076af service neutron] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:58:52.091502 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b80b94b-70ed-4ebb-b300-39d6f65076af service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:58:52.100162 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b80b94b-70ed-4ebb-b300-39d6f65076af service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:58:52.101573 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b80b94b-70ed-4ebb-b300-39d6f65076af service neutron] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:58:52.111284 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 596/1207] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 08:58:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 08:58:54.248724 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1cd18a-2909-4a7a-9f94-2bde313e6f0a None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:58:54.249275 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1cd18a-2909-4a7a-9f94-2bde313e6f0a None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:58:54.249275 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1cd18a-2909-4a7a-9f94-2bde313e6f0a None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:58:54.250562 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 612/1208] 127.0.0.1 () {54 vars in 917 bytes} [Thu Oct 20 08:58:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:58:54.255777 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-570587f0-9732-468d-88dc-1f124696f587 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:58:54.256197 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-570587f0-9732-468d-88dc-1f124696f587 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:58:54.256329 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-570587f0-9732-468d-88dc-1f124696f587 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:58:54.275467 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-570587f0-9732-468d-88dc-1f124696f587 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:58:54.276644 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-570587f0-9732-468d-88dc-1f124696f587 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:58:54.284235 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 597/1209] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Oct 20 08:58:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:58:54.576379 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1cd18a-2909-4a7a-9f94-2bde313e6f0a None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:58:54.587270 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1cd18a-2909-4a7a-9f94-2bde313e6f0a None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:58:54.591581 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-106be519-2432-4afa-a88a-a3ff3f264cf0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:58:54.592165 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-106be519-2432-4afa-a88a-a3ff3f264cf0 service neutron] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:58:54.592441 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-106be519-2432-4afa-a88a-a3ff3f264cf0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:58:54.592707 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-106be519-2432-4afa-a88a-a3ff3f264cf0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:58:54.601558 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-106be519-2432-4afa-a88a-a3ff3f264cf0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:58:54.603014 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-106be519-2432-4afa-a88a-a3ff3f264cf0 service neutron] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:58:54.613193 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 613/1210] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 08:58:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 08:58:58.230924 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0baa9423-fc0c-4611-a408-aef1e5189ea4 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:58:58.231465 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0baa9423-fc0c-4611-a408-aef1e5189ea4 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:58:58.231465 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0baa9423-fc0c-4611-a408-aef1e5189ea4 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:58:58.232732 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 598/1211] 127.0.0.1 () {54 vars in 917 bytes} [Thu Oct 20 08:58:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:58:58.238021 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7724bccf-f143-4e01-8b1e-b66df99d0710 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:58:58.238452 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7724bccf-f143-4e01-8b1e-b66df99d0710 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:58:58.238595 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7724bccf-f143-4e01-8b1e-b66df99d0710 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:58:58.258109 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-7724bccf-f143-4e01-8b1e-b66df99d0710 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:58:58.259309 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-7724bccf-f143-4e01-8b1e-b66df99d0710 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:58:58.267543 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 614/1212] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Oct 20 08:58:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:58:58.566113 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0baa9423-fc0c-4611-a408-aef1e5189ea4 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:58:58.567833 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-0baa9423-fc0c-4611-a408-aef1e5189ea4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:58:58.645172 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0baa9423-fc0c-4611-a408-aef1e5189ea4 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:58:58.648735 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81cefae1-5903-4b6d-b0ce-57497d4e6da9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:58:58.649337 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81cefae1-5903-4b6d-b0ce-57497d4e6da9 service cinder] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:58:58.649612 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81cefae1-5903-4b6d-b0ce-57497d4e6da9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:58:58.649879 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81cefae1-5903-4b6d-b0ce-57497d4e6da9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:58:58.658699 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81cefae1-5903-4b6d-b0ce-57497d4e6da9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:58:58.660163 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81cefae1-5903-4b6d-b0ce-57497d4e6da9 service cinder] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:58:58.671688 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 599/1213] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 08:58:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 08:59:00.807149 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8646cccf-d419-423e-8756-6f665c06fe4f None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:59:00.808252 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8646cccf-d419-423e-8756-6f665c06fe4f None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:59:00.808252 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8646cccf-d419-423e-8756-6f665c06fe4f None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:59:00.809586 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 615/1214] 127.0.0.1 () {54 vars in 917 bytes} [Thu Oct 20 08:59:00 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 08:59:00.814741 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a152168-6719-4528-b33b-86d83899982f None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:59:00.815169 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a152168-6719-4528-b33b-86d83899982f None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:59:00.815295 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a152168-6719-4528-b33b-86d83899982f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:59:00.834388 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-5a152168-6719-4528-b33b-86d83899982f None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 08:59:00.835600 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-5a152168-6719-4528-b33b-86d83899982f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:59:00.844056 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 600/1215] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Oct 20 08:59:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 08:59:01.147105 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8646cccf-d419-423e-8756-6f665c06fe4f None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 08:59:01.148803 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-8646cccf-d419-423e-8756-6f665c06fe4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 08:59:01.162287 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8646cccf-d419-423e-8756-6f665c06fe4f None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 08:59:01.165730 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bf056c3-d016-4a2a-b36a-139cf0266bc9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 08:59:01.166311 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf056c3-d016-4a2a-b36a-139cf0266bc9 service cinder] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 08:59:01.167271 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf056c3-d016-4a2a-b36a-139cf0266bc9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 08:59:01.167538 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf056c3-d016-4a2a-b36a-139cf0266bc9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 08:59:01.176267 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bf056c3-d016-4a2a-b36a-139cf0266bc9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 08:59:01.177700 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bf056c3-d016-4a2a-b36a-139cf0266bc9 service cinder] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 08:59:01.187212 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 616/1216] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 08:59:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 08:59:10.829751 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=88 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 08:59:11.183055 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=33 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:00:24.375127 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4f6b7c-c0fd-41bd-81c5-2c3451752be5 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:24.384162 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4f6b7c-c0fd-41bd-81c5-2c3451752be5 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:24.384162 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4f6b7c-c0fd-41bd-81c5-2c3451752be5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:24.386377 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc67916f-1e41-4a76-9da1-d3624a56f90c None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:24.389162 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc67916f-1e41-4a76-9da1-d3624a56f90c None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:24.389162 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc67916f-1e41-4a76-9da1-d3624a56f90c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:24.428699 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-1e4f6b7c-c0fd-41bd-81c5-2c3451752be5 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:00:24.430015 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-fc67916f-1e41-4a76-9da1-d3624a56f90c None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:00:24.432144 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-1e4f6b7c-c0fd-41bd-81c5-2c3451752be5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:24.433693 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-fc67916f-1e41-4a76-9da1-d3624a56f90c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:24.448387 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 617/1217] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:00:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 65 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 20 09:00:24.452316 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740f5465-02eb-4a7d-afd8-698b5f2e4cac None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:24.452580 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 601/1218] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:00:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 20 09:00:24.453089 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740f5465-02eb-4a7d-afd8-698b5f2e4cac None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:24.453703 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740f5465-02eb-4a7d-afd8-698b5f2e4cac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:24.477628 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9487886-d02c-4f96-a725-77c96d20fd31 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:24.478333 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9487886-d02c-4f96-a725-77c96d20fd31 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:24.478929 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9487886-d02c-4f96-a725-77c96d20fd31 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:24.498713 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-740f5465-02eb-4a7d-afd8-698b5f2e4cac None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:00:24.500360 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-740f5465-02eb-4a7d-afd8-698b5f2e4cac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:24.513692 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-d9487886-d02c-4f96-a725-77c96d20fd31 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:00:24.515838 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 618/1219] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:00:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 64 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:00:24.517000 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-d9487886-d02c-4f96-a725-77c96d20fd31 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:24.519397 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f9fa644-ba05-4e2e-9b5c-e973429123bb None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:24.520920 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f9fa644-ba05-4e2e-9b5c-e973429123bb None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:24.521891 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f9fa644-ba05-4e2e-9b5c-e973429123bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:24.533458 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 602/1220] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:00:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 57 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:00:24.535561 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-834b4878-f44f-47fb-9495-99e37d4dc132 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:24.536079 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-834b4878-f44f-47fb-9495-99e37d4dc132 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:24.536602 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-834b4878-f44f-47fb-9495-99e37d4dc132 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:24.548247 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-4f9fa644-ba05-4e2e-9b5c-e973429123bb None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:00:24.549441 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-4f9fa644-ba05-4e2e-9b5c-e973429123bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:24.556917 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-834b4878-f44f-47fb-9495-99e37d4dc132 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:00:24.558063 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-834b4878-f44f-47fb-9495-99e37d4dc132 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:24.558850 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 619/1221] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:00:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:00:24.562508 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5ea91d-1834-4684-b43c-c68c88aeb76b None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:24.562508 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5ea91d-1834-4684-b43c-c68c88aeb76b None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:24.562508 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5ea91d-1834-4684-b43c-c68c88aeb76b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:24.574700 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 603/1222] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:00:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:00:24.574700 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58fad27-abea-4a24-86c4-3348062715f4 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:24.574700 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58fad27-abea-4a24-86c4-3348062715f4 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:24.574700 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58fad27-abea-4a24-86c4-3348062715f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:24.580934 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-8a5ea91d-1834-4684-b43c-c68c88aeb76b None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:00:24.582236 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-8a5ea91d-1834-4684-b43c-c68c88aeb76b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:24.590600 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 620/1223] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:00:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:00:24.591650 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb342e25-c54e-40a7-b14a-18c857ac64b0 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:24.591986 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb342e25-c54e-40a7-b14a-18c857ac64b0 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:24.592260 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb342e25-c54e-40a7-b14a-18c857ac64b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:24.598419 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-d58fad27-abea-4a24-86c4-3348062715f4 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:00:24.599882 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-d58fad27-abea-4a24-86c4-3348062715f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:24.608756 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 604/1224] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:00:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:00:24.609975 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f18ecc-e83e-485d-bf1a-73b96c9bf090 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:24.610255 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f18ecc-e83e-485d-bf1a-73b96c9bf090 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:24.610547 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f18ecc-e83e-485d-bf1a-73b96c9bf090 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:24.612971 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-bb342e25-c54e-40a7-b14a-18c857ac64b0 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:00:24.614151 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-bb342e25-c54e-40a7-b14a-18c857ac64b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:24.626583 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 621/1225] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:00:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:00:24.626583 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb342e25-c54e-40a7-b14a-18c857ac64b0 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:24.637250 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb342e25-c54e-40a7-b14a-18c857ac64b0 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:24.639683 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-29f18ecc-e83e-485d-bf1a-73b96c9bf090 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:00:24.640875 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-29f18ecc-e83e-485d-bf1a-73b96c9bf090 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:24.641998 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf131970-982d-402a-a0b4-280067223357 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:24.642627 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf131970-982d-402a-a0b4-280067223357 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:24.642901 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf131970-982d-402a-a0b4-280067223357 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:24.643168 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf131970-982d-402a-a0b4-280067223357 admin admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:24.643805 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf131970-982d-402a-a0b4-280067223357 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 09:00:24.644102 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf131970-982d-402a-a0b4-280067223357 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:24.645705 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf131970-982d-402a-a0b4-280067223357 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:24.648722 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 605/1226] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:00:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:00:24.649494 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29f18ecc-e83e-485d-bf1a-73b96c9bf090 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:24.658945 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29f18ecc-e83e-485d-bf1a-73b96c9bf090 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:24.659098 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 622/1227] 10.222.0.36 () {58 vars in 1153 bytes} [Thu Oct 20 09:00:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 09:00:24.660413 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d208791-d56e-44fb-b81c-b5c59969ba12 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:24.660702 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d208791-d56e-44fb-b81c-b5c59969ba12 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:24.660971 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d208791-d56e-44fb-b81c-b5c59969ba12 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:24.663785 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c83c0d7f-66c3-47c8-98a2-56f74537286f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:24.664469 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c83c0d7f-66c3-47c8-98a2-56f74537286f admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:24.664750 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c83c0d7f-66c3-47c8-98a2-56f74537286f admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:24.665019 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c83c0d7f-66c3-47c8-98a2-56f74537286f admin admin] PATH_INFO: `/v3/domains` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:24.666199 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c83c0d7f-66c3-47c8-98a2-56f74537286f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 09:00:24.666530 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c83c0d7f-66c3-47c8-98a2-56f74537286f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:24.667804 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: WARNING py.warnings [None req-c83c0d7f-66c3-47c8-98a2-56f74537286f admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 20 09:00:24.667804 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: warnings.warn(msg) Oct 20 09:00:24.667804 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: Oct 20 09:00:24.668712 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c83c0d7f-66c3-47c8-98a2-56f74537286f admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:24.687921 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 606/1228] 10.222.0.36 () {58 vars in 1153 bytes} [Thu Oct 20 09:00:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 09:00:24.688125 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-7d208791-d56e-44fb-b81c-b5c59969ba12 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:00:24.688217 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b17ca356-29ff-4c99-aab9-089f2377928d None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:24.688217 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b17ca356-29ff-4c99-aab9-089f2377928d None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:24.688424 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-7d208791-d56e-44fb-b81c-b5c59969ba12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:24.688525 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b17ca356-29ff-4c99-aab9-089f2377928d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:24.690869 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 623/1229] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:00:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:00:24.691646 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d208791-d56e-44fb-b81c-b5c59969ba12 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:24.701078 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d208791-d56e-44fb-b81c-b5c59969ba12 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:24.702778 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-b17ca356-29ff-4c99-aab9-089f2377928d None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:00:24.704100 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-b17ca356-29ff-4c99-aab9-089f2377928d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:24.705569 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b8fb624-71d2-4b58-97a6-7fc9f02e2bb3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:24.706144 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8fb624-71d2-4b58-97a6-7fc9f02e2bb3 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:24.706420 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8fb624-71d2-4b58-97a6-7fc9f02e2bb3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:24.706725 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8fb624-71d2-4b58-97a6-7fc9f02e2bb3 admin admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:24.707467 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b8fb624-71d2-4b58-97a6-7fc9f02e2bb3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 09:00:24.707779 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b8fb624-71d2-4b58-97a6-7fc9f02e2bb3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:24.709358 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b8fb624-71d2-4b58-97a6-7fc9f02e2bb3 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:24.712099 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 607/1230] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:00:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:00:24.715894 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de879625-88b4-4181-aa2a-f39deda5ddd0 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:24.715894 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de879625-88b4-4181-aa2a-f39deda5ddd0 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:24.715894 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de879625-88b4-4181-aa2a-f39deda5ddd0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:24.721516 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 624/1231] 10.222.0.36 () {58 vars in 1153 bytes} [Thu Oct 20 09:00:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 09:00:24.722724 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff19799-1bb1-41f3-aec1-fafd970baa9f None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:24.723007 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff19799-1bb1-41f3-aec1-fafd970baa9f None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:24.723278 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eff19799-1bb1-41f3-aec1-fafd970baa9f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:24.734336 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-de879625-88b4-4181-aa2a-f39deda5ddd0 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:00:24.735539 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-de879625-88b4-4181-aa2a-f39deda5ddd0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:24.742985 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-eff19799-1bb1-41f3-aec1-fafd970baa9f None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:00:24.743344 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 608/1232] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:00:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:00:24.744855 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-eff19799-1bb1-41f3-aec1-fafd970baa9f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:24.745116 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e96634-64f1-488c-9eaa-6f86f938d391 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:24.745400 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e96634-64f1-488c-9eaa-6f86f938d391 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:24.745711 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e96634-64f1-488c-9eaa-6f86f938d391 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:24.752821 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 625/1233] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:00:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:00:24.753872 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eff19799-1bb1-41f3-aec1-fafd970baa9f None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:24.764652 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eff19799-1bb1-41f3-aec1-fafd970baa9f None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:24.765493 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-c5e96634-64f1-488c-9eaa-6f86f938d391 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:00:24.766716 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-c5e96634-64f1-488c-9eaa-6f86f938d391 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:24.769216 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79031ccc-a1f8-42eb-903e-d44ec174f50a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:24.769807 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79031ccc-a1f8-42eb-903e-d44ec174f50a admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:24.770078 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79031ccc-a1f8-42eb-903e-d44ec174f50a admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:24.770344 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79031ccc-a1f8-42eb-903e-d44ec174f50a admin admin] PATH_INFO: `/v3/projects` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:24.771056 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79031ccc-a1f8-42eb-903e-d44ec174f50a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:24.772086 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: WARNING py.warnings [None req-79031ccc-a1f8-42eb-903e-d44ec174f50a admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 20 09:00:24.772086 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: warnings.warn(msg) Oct 20 09:00:24.772086 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: Oct 20 09:00:24.773266 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79031ccc-a1f8-42eb-903e-d44ec174f50a admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:24.774752 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 609/1234] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:00:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:00:24.775532 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5e96634-64f1-488c-9eaa-6f86f938d391 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:24.785099 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5e96634-64f1-488c-9eaa-6f86f938d391 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:24.789815 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca3e7248-5259-4419-94ee-cd1c81cab53a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:24.790426 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3e7248-5259-4419-94ee-cd1c81cab53a admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:24.791266 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3e7248-5259-4419-94ee-cd1c81cab53a admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:24.791534 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3e7248-5259-4419-94ee-cd1c81cab53a admin admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:24.792151 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca3e7248-5259-4419-94ee-cd1c81cab53a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:24.794065 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca3e7248-5259-4419-94ee-cd1c81cab53a admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:24.811375 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 626/1235] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:00:24 2022] POST /identity/v3/projects => generated 393 bytes in 58 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:00:24.812245 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79031ccc-a1f8-42eb-903e-d44ec174f50a admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:24.822952 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 610/1236] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:00:24 2022] POST /identity/v3/projects => generated 387 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:00:24.823779 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca3e7248-5259-4419-94ee-cd1c81cab53a admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:24.873469 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79031ccc-a1f8-42eb-903e-d44ec174f50a admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:24.889422 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca3e7248-5259-4419-94ee-cd1c81cab53a admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:24.893967 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87a9e5c2-50c0-42b6-8365-641574af580e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:24.894605 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a9e5c2-50c0-42b6-8365-641574af580e admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:24.894887 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a9e5c2-50c0-42b6-8365-641574af580e admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:24.895159 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a9e5c2-50c0-42b6-8365-641574af580e admin admin] PATH_INFO: `/v3/domains` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:24.895793 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87a9e5c2-50c0-42b6-8365-641574af580e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 09:00:24.896094 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87a9e5c2-50c0-42b6-8365-641574af580e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:24.897690 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87a9e5c2-50c0-42b6-8365-641574af580e admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:24.909042 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 611/1237] 10.222.0.36 () {58 vars in 1153 bytes} [Thu Oct 20 09:00:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 09:00:24.917016 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87a9e5c2-50c0-42b6-8365-641574af580e admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:24.926099 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87a9e5c2-50c0-42b6-8365-641574af580e admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:24.928129 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-848c2f48-106d-4e0b-9247-959db02dcdc7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:24.928739 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-848c2f48-106d-4e0b-9247-959db02dcdc7 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:24.929016 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-848c2f48-106d-4e0b-9247-959db02dcdc7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:24.929284 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-848c2f48-106d-4e0b-9247-959db02dcdc7 admin admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:24.929922 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-848c2f48-106d-4e0b-9247-959db02dcdc7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 09:00:24.930219 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-848c2f48-106d-4e0b-9247-959db02dcdc7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:24.930973 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c29f820a-e387-4646-8282-b929a842833b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:24.932113 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c29f820a-e387-4646-8282-b929a842833b admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:24.932391 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c29f820a-e387-4646-8282-b929a842833b admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:24.932662 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c29f820a-e387-4646-8282-b929a842833b admin admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:24.933091 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-848c2f48-106d-4e0b-9247-959db02dcdc7 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:24.933497 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c29f820a-e387-4646-8282-b929a842833b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:24.935422 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c29f820a-e387-4646-8282-b929a842833b admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:24.939992 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:00:24.945970 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 627/1238] 10.222.0.36 () {58 vars in 1153 bytes} [Thu Oct 20 09:00:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 09:00:24.947103 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-848c2f48-106d-4e0b-9247-959db02dcdc7 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:24.958208 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-848c2f48-106d-4e0b-9247-959db02dcdc7 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:24.963054 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-643804eb-9296-4231-9765-3f3e7a18684f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:24.963676 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-643804eb-9296-4231-9765-3f3e7a18684f admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:24.963957 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-643804eb-9296-4231-9765-3f3e7a18684f admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:24.964234 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-643804eb-9296-4231-9765-3f3e7a18684f admin admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:24.965127 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-643804eb-9296-4231-9765-3f3e7a18684f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 09:00:24.965426 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-643804eb-9296-4231-9765-3f3e7a18684f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:24.966046 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 612/1239] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:00:24 2022] POST /identity/v3/projects => generated 379 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:00:24.967049 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-643804eb-9296-4231-9765-3f3e7a18684f admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:24.967311 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79268730-462b-4fa5-b996-64cf167da1e9 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:24.967586 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79268730-462b-4fa5-b996-64cf167da1e9 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:24.967853 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79268730-462b-4fa5-b996-64cf167da1e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:24.986631 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 628/1240] 10.222.0.36 () {58 vars in 1153 bytes} [Thu Oct 20 09:00:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 09:00:24.986631 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-643804eb-9296-4231-9765-3f3e7a18684f admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:24.988213 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-79268730-462b-4fa5-b996-64cf167da1e9 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:00:24.989390 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-79268730-462b-4fa5-b996-64cf167da1e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:24.996847 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-643804eb-9296-4231-9765-3f3e7a18684f admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:25.031159 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:00:25.051857 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 613/1241] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:00:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:00:25.052188 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf5456f1-37f9-459c-b200-dab1f762f1b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:25.052725 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79268730-462b-4fa5-b996-64cf167da1e9 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:25.052857 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf5456f1-37f9-459c-b200-dab1f762f1b0 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:25.053024 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf5456f1-37f9-459c-b200-dab1f762f1b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:25.053268 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf5456f1-37f9-459c-b200-dab1f762f1b0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:25.053918 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf5456f1-37f9-459c-b200-dab1f762f1b0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 09:00:25.054221 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf5456f1-37f9-459c-b200-dab1f762f1b0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:25.055854 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf5456f1-37f9-459c-b200-dab1f762f1b0 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:25.064640 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79268730-462b-4fa5-b996-64cf167da1e9 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:25.074284 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 629/1242] 10.222.0.36 () {58 vars in 1153 bytes} [Thu Oct 20 09:00:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 09:00:25.074284 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf5456f1-37f9-459c-b200-dab1f762f1b0 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:25.074567 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6498fd28-f208-47b2-a0c2-7e0164e4a69b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:25.074567 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6498fd28-f208-47b2-a0c2-7e0164e4a69b admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:25.074567 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6498fd28-f208-47b2-a0c2-7e0164e4a69b admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:25.074567 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6498fd28-f208-47b2-a0c2-7e0164e4a69b admin admin] PATH_INFO: `/v3/users` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:25.074896 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6498fd28-f208-47b2-a0c2-7e0164e4a69b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:25.074896 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: WARNING py.warnings [None req-6498fd28-f208-47b2-a0c2-7e0164e4a69b admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 20 09:00:25.074896 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: warnings.warn(msg) Oct 20 09:00:25.074896 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: Oct 20 09:00:25.074896 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6498fd28-f208-47b2-a0c2-7e0164e4a69b admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:25.077695 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf5456f1-37f9-459c-b200-dab1f762f1b0 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:25.082401 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0077cd87-c476-4bcf-983f-c98d1c04c2d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:25.083010 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0077cd87-c476-4bcf-983f-c98d1c04c2d4 admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:25.083290 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0077cd87-c476-4bcf-983f-c98d1c04c2d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:25.083560 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0077cd87-c476-4bcf-983f-c98d1c04c2d4 admin admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:25.084174 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0077cd87-c476-4bcf-983f-c98d1c04c2d4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:25.086084 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0077cd87-c476-4bcf-983f-c98d1c04c2d4 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:25.095705 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 614/1243] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:00:25 2022] POST /identity/v3/users => generated 297 bytes in 43 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:00:25.095705 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6498fd28-f208-47b2-a0c2-7e0164e4a69b admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:25.108726 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6498fd28-f208-47b2-a0c2-7e0164e4a69b admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:25.112271 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 630/1244] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:00:25 2022] POST /identity/v3/users => generated 294 bytes in 44 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:00:25.112919 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0077cd87-c476-4bcf-983f-c98d1c04c2d4 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:25.113638 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46277d44-93aa-43ff-8455-d17a9c239b57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:25.114366 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46277d44-93aa-43ff-8455-d17a9c239b57 admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:25.114677 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46277d44-93aa-43ff-8455-d17a9c239b57 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:25.115131 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46277d44-93aa-43ff-8455-d17a9c239b57 admin admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:25.115640 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46277d44-93aa-43ff-8455-d17a9c239b57 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:25.117545 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46277d44-93aa-43ff-8455-d17a9c239b57 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:25.122138 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0077cd87-c476-4bcf-983f-c98d1c04c2d4 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:25.126651 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11bf2e3d-e382-49f3-9d71-f205254c6ee4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:25.127227 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11bf2e3d-e382-49f3-9d71-f205254c6ee4 admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:25.127505 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11bf2e3d-e382-49f3-9d71-f205254c6ee4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:25.127777 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11bf2e3d-e382-49f3-9d71-f205254c6ee4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:25.128387 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11bf2e3d-e382-49f3-9d71-f205254c6ee4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:25.130583 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11bf2e3d-e382-49f3-9d71-f205254c6ee4 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:25.147599 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 615/1245] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:00:25 2022] POST /identity/v3/projects => generated 385 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:00:25.148482 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46277d44-93aa-43ff-8455-d17a9c239b57 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:25.166179 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 631/1246] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:00:25 2022] POST /identity/v3/projects => generated 371 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:00:25.167580 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11bf2e3d-e382-49f3-9d71-f205254c6ee4 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:25.209199 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46277d44-93aa-43ff-8455-d17a9c239b57 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:25.229431 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11bf2e3d-e382-49f3-9d71-f205254c6ee4 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:25.233981 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd4b6b9-8fe9-4c85-bc63-2df6d6ce68b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:25.234616 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd4b6b9-8fe9-4c85-bc63-2df6d6ce68b9 admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:25.234892 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd4b6b9-8fe9-4c85-bc63-2df6d6ce68b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:25.235155 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd4b6b9-8fe9-4c85-bc63-2df6d6ce68b9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:25.235772 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cd4b6b9-8fe9-4c85-bc63-2df6d6ce68b9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:25.237679 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cd4b6b9-8fe9-4c85-bc63-2df6d6ce68b9 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:25.263933 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 632/1247] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:00:25 2022] POST /identity/v3/projects => generated 365 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:00:25.264756 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd4b6b9-8fe9-4c85-bc63-2df6d6ce68b9 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:25.267851 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0eba7fb-d4ff-46e5-9fb4-f0e7dd6e347e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:25.268753 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0eba7fb-d4ff-46e5-9fb4-f0e7dd6e347e admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:25.269025 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0eba7fb-d4ff-46e5-9fb4-f0e7dd6e347e admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:25.269530 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0eba7fb-d4ff-46e5-9fb4-f0e7dd6e347e admin admin] PATH_INFO: `/v3/users` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:25.270168 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0eba7fb-d4ff-46e5-9fb4-f0e7dd6e347e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:25.272184 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0eba7fb-d4ff-46e5-9fb4-f0e7dd6e347e admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:25.293221 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:00:25.295468 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 616/1248] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:00:25 2022] POST /identity/v3/users => generated 290 bytes in 147 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:00:25.296321 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0eba7fb-d4ff-46e5-9fb4-f0e7dd6e347e admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:25.306098 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0eba7fb-d4ff-46e5-9fb4-f0e7dd6e347e admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:25.330901 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd4b6b9-8fe9-4c85-bc63-2df6d6ce68b9 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:25.335553 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-690c2391-78c7-459f-973e-fd6a53b23dff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:25.336151 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690c2391-78c7-459f-973e-fd6a53b23dff admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:25.336430 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690c2391-78c7-459f-973e-fd6a53b23dff admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:25.336705 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690c2391-78c7-459f-973e-fd6a53b23dff admin admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:25.337866 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-690c2391-78c7-459f-973e-fd6a53b23dff admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 09:00:25.338171 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-690c2391-78c7-459f-973e-fd6a53b23dff admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:25.340070 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-690c2391-78c7-459f-973e-fd6a53b23dff admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:25.346786 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:00:25.352611 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 633/1249] 10.222.0.36 () {58 vars in 1153 bytes} [Thu Oct 20 09:00:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 09:00:25.360530 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-690c2391-78c7-459f-973e-fd6a53b23dff admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:25.365131 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae1a9a5d-a32f-4146-8e80-749e1c245711 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:25.365331 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-690c2391-78c7-459f-973e-fd6a53b23dff admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:25.365747 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae1a9a5d-a32f-4146-8e80-749e1c245711 admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:25.366024 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae1a9a5d-a32f-4146-8e80-749e1c245711 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:25.366307 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae1a9a5d-a32f-4146-8e80-749e1c245711 admin admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:25.366960 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae1a9a5d-a32f-4146-8e80-749e1c245711 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:25.368941 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae1a9a5d-a32f-4146-8e80-749e1c245711 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:25.370010 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0483c7e-6f62-441e-9147-15fe9b3bd2ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:25.370630 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0483c7e-6f62-441e-9147-15fe9b3bd2ae admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:25.370907 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0483c7e-6f62-441e-9147-15fe9b3bd2ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:25.371174 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0483c7e-6f62-441e-9147-15fe9b3bd2ae admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:25.371837 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0483c7e-6f62-441e-9147-15fe9b3bd2ae admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:25.373532 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0483c7e-6f62-441e-9147-15fe9b3bd2ae admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:25.383548 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 634/1250] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:00:25 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:00:25.388112 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0483c7e-6f62-441e-9147-15fe9b3bd2ae admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:25.397761 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0483c7e-6f62-441e-9147-15fe9b3bd2ae admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:25.397915 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 617/1251] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:00:25 2022] POST /identity/v3/projects => generated 357 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:00:25.398668 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae1a9a5d-a32f-4146-8e80-749e1c245711 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:25.456744 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-423f10ca-1060-42f1-a6df-e467edd3971e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:25.457348 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423f10ca-1060-42f1-a6df-e467edd3971e admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:25.457621 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423f10ca-1060-42f1-a6df-e467edd3971e admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:25.457887 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423f10ca-1060-42f1-a6df-e467edd3971e admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:25.458557 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-423f10ca-1060-42f1-a6df-e467edd3971e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:25.460148 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-423f10ca-1060-42f1-a6df-e467edd3971e admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:25.461991 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae1a9a5d-a32f-4146-8e80-749e1c245711 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:25.468050 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c19a2be-8c77-4bb6-9a00-b230f17782ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:25.468691 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c19a2be-8c77-4bb6-9a00-b230f17782ed admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:25.469018 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c19a2be-8c77-4bb6-9a00-b230f17782ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:25.469350 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c19a2be-8c77-4bb6-9a00-b230f17782ed admin admin] PATH_INFO: `/v3/users` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:25.470042 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c19a2be-8c77-4bb6-9a00-b230f17782ed admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:25.471261 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 635/1252] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:00:25 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:00:25.472040 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-423f10ca-1060-42f1-a6df-e467edd3971e admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:25.473456 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c19a2be-8c77-4bb6-9a00-b230f17782ed admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:25.481181 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-423f10ca-1060-42f1-a6df-e467edd3971e admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:25.485960 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48693e36-4d02-455b-9c6a-62fdc484f274 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:25.486641 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48693e36-4d02-455b-9c6a-62fdc484f274 admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:25.486964 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48693e36-4d02-455b-9c6a-62fdc484f274 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:25.487267 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48693e36-4d02-455b-9c6a-62fdc484f274 admin admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:25.487964 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48693e36-4d02-455b-9c6a-62fdc484f274 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:25.490433 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48693e36-4d02-455b-9c6a-62fdc484f274 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:25.496753 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 618/1253] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:00:25 2022] POST /identity/v3/users => generated 293 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:00:25.497616 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c19a2be-8c77-4bb6-9a00-b230f17782ed admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:25.507654 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c19a2be-8c77-4bb6-9a00-b230f17782ed admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:25.512941 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba2615d-a8ce-4c0c-864d-9244569373af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:25.513557 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba2615d-a8ce-4c0c-864d-9244569373af admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:25.513860 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba2615d-a8ce-4c0c-864d-9244569373af admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:25.514131 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba2615d-a8ce-4c0c-864d-9244569373af admin admin] PATH_INFO: `/v3/users` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:25.514389 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 636/1254] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:00:25 2022] POST /identity/v3/users => generated 286 bytes in 42 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:00:25.514778 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ba2615d-a8ce-4c0c-864d-9244569373af admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:25.515184 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48693e36-4d02-455b-9c6a-62fdc484f274 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:25.516726 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ba2615d-a8ce-4c0c-864d-9244569373af admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:25.524564 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48693e36-4d02-455b-9c6a-62fdc484f274 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:25.529855 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac78082f-a37d-4a2c-bc0d-43b6cea185e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:25.530589 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac78082f-a37d-4a2c-bc0d-43b6cea185e6 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:25.530926 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac78082f-a37d-4a2c-bc0d-43b6cea185e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:25.531214 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac78082f-a37d-4a2c-bc0d-43b6cea185e6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:25.532032 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac78082f-a37d-4a2c-bc0d-43b6cea185e6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:25.534231 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac78082f-a37d-4a2c-bc0d-43b6cea185e6 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:25.542012 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 619/1255] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:00:25 2022] POST /identity/v3/users => generated 283 bytes in 42 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:00:25.542012 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba2615d-a8ce-4c0c-864d-9244569373af admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:25.550211 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 637/1256] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:00:25 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:00:25.550211 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac78082f-a37d-4a2c-bc0d-43b6cea185e6 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:25.550447 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba2615d-a8ce-4c0c-864d-9244569373af admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:25.553592 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0beda8d-2ce3-41d6-919a-5202f3f53d4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:25.554174 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0beda8d-2ce3-41d6-919a-5202f3f53d4b admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:25.554450 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0beda8d-2ce3-41d6-919a-5202f3f53d4b admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:25.554746 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0beda8d-2ce3-41d6-919a-5202f3f53d4b admin admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:25.555363 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0beda8d-2ce3-41d6-919a-5202f3f53d4b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:25.557258 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0beda8d-2ce3-41d6-919a-5202f3f53d4b admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:25.557922 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac78082f-a37d-4a2c-bc0d-43b6cea185e6 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:25.562599 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03f439e6-d085-4d42-bed5-c7b365e1795a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:25.563170 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f439e6-d085-4d42-bed5-c7b365e1795a admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:25.563439 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f439e6-d085-4d42-bed5-c7b365e1795a admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:25.563703 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f439e6-d085-4d42-bed5-c7b365e1795a admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:25.564411 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03f439e6-d085-4d42-bed5-c7b365e1795a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:25.565990 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03f439e6-d085-4d42-bed5-c7b365e1795a admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:25.576568 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 638/1257] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:00:25 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:00:25.577497 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03f439e6-d085-4d42-bed5-c7b365e1795a admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:25.586719 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 620/1258] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:00:25 2022] POST /identity/v3/projects => generated 385 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:00:25.588000 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03f439e6-d085-4d42-bed5-c7b365e1795a admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:25.588149 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0beda8d-2ce3-41d6-919a-5202f3f53d4b admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:25.645165 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83cce8ba-362c-4ac6-ad1a-98b3d0a21dd5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:25.645767 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83cce8ba-362c-4ac6-ad1a-98b3d0a21dd5 admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:25.646049 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83cce8ba-362c-4ac6-ad1a-98b3d0a21dd5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:25.646318 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83cce8ba-362c-4ac6-ad1a-98b3d0a21dd5 admin admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:25.646979 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83cce8ba-362c-4ac6-ad1a-98b3d0a21dd5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:25.648998 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83cce8ba-362c-4ac6-ad1a-98b3d0a21dd5 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:25.649635 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0beda8d-2ce3-41d6-919a-5202f3f53d4b admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:25.654831 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4648d743-d1dc-43ad-862d-8911513e447f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:25.655455 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4648d743-d1dc-43ad-862d-8911513e447f admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:25.655731 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4648d743-d1dc-43ad-862d-8911513e447f admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:25.656001 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4648d743-d1dc-43ad-862d-8911513e447f admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:25.656637 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4648d743-d1dc-43ad-862d-8911513e447f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:25.658247 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4648d743-d1dc-43ad-862d-8911513e447f admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:25.659850 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:00:25.669263 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 621/1259] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:00:25 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:00:25.670300 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4648d743-d1dc-43ad-862d-8911513e447f admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:25.675780 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 639/1260] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:00:25 2022] POST /identity/v3/users => generated 279 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:00:25.679440 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83cce8ba-362c-4ac6-ad1a-98b3d0a21dd5 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:25.680402 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4648d743-d1dc-43ad-862d-8911513e447f admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:25.684936 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa8c0213-ffb8-4edb-915c-906c977e1064 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:25.685516 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8c0213-ffb8-4edb-915c-906c977e1064 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:25.685792 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8c0213-ffb8-4edb-915c-906c977e1064 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:25.686060 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8c0213-ffb8-4edb-915c-906c977e1064 admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:25.686714 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa8c0213-ffb8-4edb-915c-906c977e1064 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:25.688301 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa8c0213-ffb8-4edb-915c-906c977e1064 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:25.688993 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83cce8ba-362c-4ac6-ad1a-98b3d0a21dd5 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:25.693507 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdbe542b-30e0-4167-ac85-b1be31e8ed90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:25.694082 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdbe542b-30e0-4167-ac85-b1be31e8ed90 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:25.694359 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdbe542b-30e0-4167-ac85-b1be31e8ed90 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:25.694660 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdbe542b-30e0-4167-ac85-b1be31e8ed90 admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:25.695285 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdbe542b-30e0-4167-ac85-b1be31e8ed90 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:25.697549 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdbe542b-30e0-4167-ac85-b1be31e8ed90 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:25.698694 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 622/1261] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:00:25 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:00:25.699637 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa8c0213-ffb8-4edb-915c-906c977e1064 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:25.704898 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:00:25.709217 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 640/1262] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:00:25 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:00:25.710096 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdbe542b-30e0-4167-ac85-b1be31e8ed90 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:25.710345 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa8c0213-ffb8-4edb-915c-906c977e1064 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:25.717267 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-046018f9-c44c-4ddd-8e0c-2b16f6dcfcec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:25.717876 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046018f9-c44c-4ddd-8e0c-2b16f6dcfcec admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:25.718352 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046018f9-c44c-4ddd-8e0c-2b16f6dcfcec admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:25.718659 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046018f9-c44c-4ddd-8e0c-2b16f6dcfcec admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:25.719347 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-046018f9-c44c-4ddd-8e0c-2b16f6dcfcec admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:25.720724 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdbe542b-30e0-4167-ac85-b1be31e8ed90 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:25.721004 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-046018f9-c44c-4ddd-8e0c-2b16f6dcfcec admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:25.725213 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07477898-026c-4077-b9b0-479cff34fb68 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:25.725776 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07477898-026c-4077-b9b0-479cff34fb68 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:25.726105 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07477898-026c-4077-b9b0-479cff34fb68 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:25.726377 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07477898-026c-4077-b9b0-479cff34fb68 admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:25.727039 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07477898-026c-4077-b9b0-479cff34fb68 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:25.728642 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07477898-026c-4077-b9b0-479cff34fb68 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:25.731934 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 623/1263] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:00:25 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:00:25.732737 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-046018f9-c44c-4ddd-8e0c-2b16f6dcfcec admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:25.739771 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 641/1264] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:00:25 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:00:25.740846 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07477898-026c-4077-b9b0-479cff34fb68 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:25.742461 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-046018f9-c44c-4ddd-8e0c-2b16f6dcfcec admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:25.747736 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-223ed130-d6cb-49b0-98a7-4e4408687a1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:25.748875 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223ed130-d6cb-49b0-98a7-4e4408687a1d admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:25.749158 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223ed130-d6cb-49b0-98a7-4e4408687a1d admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:25.749432 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223ed130-d6cb-49b0-98a7-4e4408687a1d admin admin] PATH_INFO: `/v3/projects/d24a1fc0e9344af181d49cfc58f0c88b/users/7f932695a5994c5790d14c210ea65326/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:25.751661 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07477898-026c-4077-b9b0-479cff34fb68 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:25.756328 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8417ac6f-86a8-4b58-ad7a-5a85d8521456 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:25.756914 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8417ac6f-86a8-4b58-ad7a-5a85d8521456 admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:25.757174 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8417ac6f-86a8-4b58-ad7a-5a85d8521456 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:25.757447 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8417ac6f-86a8-4b58-ad7a-5a85d8521456 admin admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:25.758056 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8417ac6f-86a8-4b58-ad7a-5a85d8521456 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:25.760828 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8417ac6f-86a8-4b58-ad7a-5a85d8521456 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:25.765086 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-223ed130-d6cb-49b0-98a7-4e4408687a1d admin admin] RBAC: Authorizing `identity:create_grant(project_id=d24a1fc0e9344af181d49cfc58f0c88b, user_id=7f932695a5994c5790d14c210ea65326, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:25.767829 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-223ed130-d6cb-49b0-98a7-4e4408687a1d admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:25.781485 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 624/1265] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:00:25 2022] PUT /identity/v3/projects/d24a1fc0e9344af181d49cfc58f0c88b/users/7f932695a5994c5790d14c210ea65326/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:00:25.782266 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-223ed130-d6cb-49b0-98a7-4e4408687a1d admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:25.784730 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 642/1266] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:00:25 2022] POST /identity/v3/users => generated 293 bytes in 44 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:00:25.785407 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8417ac6f-86a8-4b58-ad7a-5a85d8521456 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:25.796129 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8417ac6f-86a8-4b58-ad7a-5a85d8521456 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:25.845943 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-223ed130-d6cb-49b0-98a7-4e4408687a1d admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:25.850854 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63e238c7-4439-42f1-9d0a-e7448e7c30c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:25.851514 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e238c7-4439-42f1-9d0a-e7448e7c30c7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:25.851794 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e238c7-4439-42f1-9d0a-e7448e7c30c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:25.852067 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e238c7-4439-42f1-9d0a-e7448e7c30c7 admin admin] PATH_INFO: `/v3/projects/869b5cf3a9264fd3b70ae656d7814cfb/users/fd9733d1bb2b4520b70090542615ddf9/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:25.853920 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1cdf47-1b39-44f0-bc71-8ef4c0323d81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:25.855775 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1cdf47-1b39-44f0-bc71-8ef4c0323d81 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:25.856070 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1cdf47-1b39-44f0-bc71-8ef4c0323d81 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:25.856351 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1cdf47-1b39-44f0-bc71-8ef4c0323d81 admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:25.857026 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d1cdf47-1b39-44f0-bc71-8ef4c0323d81 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:25.858689 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d1cdf47-1b39-44f0-bc71-8ef4c0323d81 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:25.867238 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63e238c7-4439-42f1-9d0a-e7448e7c30c7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=869b5cf3a9264fd3b70ae656d7814cfb, user_id=fd9733d1bb2b4520b70090542615ddf9, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:25.868494 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 643/1267] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:00:25 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:00:25.869834 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1cdf47-1b39-44f0-bc71-8ef4c0323d81 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:25.871026 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63e238c7-4439-42f1-9d0a-e7448e7c30c7 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:25.879884 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1cdf47-1b39-44f0-bc71-8ef4c0323d81 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:25.885233 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 625/1268] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:00:25 2022] PUT /identity/v3/projects/869b5cf3a9264fd3b70ae656d7814cfb/users/fd9733d1bb2b4520b70090542615ddf9/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:00:25.886007 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63e238c7-4439-42f1-9d0a-e7448e7c30c7 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:25.938946 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bd258a5-939d-4ee0-ad3e-4a34ed6e96ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:25.939489 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd258a5-939d-4ee0-ad3e-4a34ed6e96ba admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:25.939804 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd258a5-939d-4ee0-ad3e-4a34ed6e96ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:25.940113 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd258a5-939d-4ee0-ad3e-4a34ed6e96ba admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:25.940770 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bd258a5-939d-4ee0-ad3e-4a34ed6e96ba admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:25.942512 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bd258a5-939d-4ee0-ad3e-4a34ed6e96ba admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:25.945890 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63e238c7-4439-42f1-9d0a-e7448e7c30c7 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:25.951034 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ac4a8c4-af60-43a3-a176-91729838ff1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:25.951625 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac4a8c4-af60-43a3-a176-91729838ff1a admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:25.951899 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac4a8c4-af60-43a3-a176-91729838ff1a admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:25.952168 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac4a8c4-af60-43a3-a176-91729838ff1a admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:25.952793 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ac4a8c4-af60-43a3-a176-91729838ff1a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:25.952917 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 644/1269] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:00:25 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:00:25.953599 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bd258a5-939d-4ee0-ad3e-4a34ed6e96ba admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:25.954450 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ac4a8c4-af60-43a3-a176-91729838ff1a admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:25.963129 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bd258a5-939d-4ee0-ad3e-4a34ed6e96ba admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:25.967925 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 626/1270] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:00:25 2022] GET /identity/v3/roles => generated 1420 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:00:25.967925 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ac4a8c4-af60-43a3-a176-91729838ff1a admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:25.968236 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39521f9f-3171-4fd9-8988-75d0fc638fdf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:25.968818 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39521f9f-3171-4fd9-8988-75d0fc638fdf admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:25.969103 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39521f9f-3171-4fd9-8988-75d0fc638fdf admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:25.969376 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39521f9f-3171-4fd9-8988-75d0fc638fdf admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:25.970003 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39521f9f-3171-4fd9-8988-75d0fc638fdf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:25.971607 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39521f9f-3171-4fd9-8988-75d0fc638fdf admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:25.976475 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ac4a8c4-af60-43a3-a176-91729838ff1a admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:25.980892 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d81f82f3-32ed-4550-8f94-028ef4443834 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:25.981499 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d81f82f3-32ed-4550-8f94-028ef4443834 admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:25.981773 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d81f82f3-32ed-4550-8f94-028ef4443834 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:25.982256 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d81f82f3-32ed-4550-8f94-028ef4443834 admin admin] PATH_INFO: `/v3/projects/4da29c9977ca44baa867b3c7fc86164b/users/c7c40a4592874732871eaabd04d8662c/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:25.982393 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 645/1271] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:00:25 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:00:25.983304 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-910bea08-af2e-4fcc-99ab-122080356861 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:25.983586 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-910bea08-af2e-4fcc-99ab-122080356861 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:25.983860 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-910bea08-af2e-4fcc-99ab-122080356861 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:25.998431 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d81f82f3-32ed-4550-8f94-028ef4443834 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4da29c9977ca44baa867b3c7fc86164b, user_id=c7c40a4592874732871eaabd04d8662c, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:26.001114 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d81f82f3-32ed-4550-8f94-028ef4443834 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:26.002756 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-910bea08-af2e-4fcc-99ab-122080356861 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:00:26.003965 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-910bea08-af2e-4fcc-99ab-122080356861 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:26.014505 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 646/1272] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:00:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:00:26.014505 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-910bea08-af2e-4fcc-99ab-122080356861 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:26.019183 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 627/1273] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:00:25 2022] PUT /identity/v3/projects/4da29c9977ca44baa867b3c7fc86164b/users/c7c40a4592874732871eaabd04d8662c/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:00:26.020171 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d81f82f3-32ed-4550-8f94-028ef4443834 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:26.026623 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-910bea08-af2e-4fcc-99ab-122080356861 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:26.048223 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:00:26.052986 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:INSERT=4,keystone:SELECT=96 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:00:26.084380 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f21cce8-e586-42e6-95da-533685b448bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:26.085054 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f21cce8-e586-42e6-95da-533685b448bc admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:26.085262 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f21cce8-e586-42e6-95da-533685b448bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:26.085529 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f21cce8-e586-42e6-95da-533685b448bc admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:26.086168 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f21cce8-e586-42e6-95da-533685b448bc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:26.087798 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f21cce8-e586-42e6-95da-533685b448bc admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:26.088360 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d81f82f3-32ed-4550-8f94-028ef4443834 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:26.093212 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c5e2e07-a652-4a05-803e-4f1d7eb05265 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:26.093805 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5e2e07-a652-4a05-803e-4f1d7eb05265 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:26.094080 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5e2e07-a652-4a05-803e-4f1d7eb05265 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:26.094349 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5e2e07-a652-4a05-803e-4f1d7eb05265 admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:26.095012 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c5e2e07-a652-4a05-803e-4f1d7eb05265 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:26.096589 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c5e2e07-a652-4a05-803e-4f1d7eb05265 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:26.097945 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 647/1274] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:00:26 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:00:26.099188 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b0b7bc-1392-430a-9b86-7e9989679f14 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:26.099469 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b0b7bc-1392-430a-9b86-7e9989679f14 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:26.099732 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b0b7bc-1392-430a-9b86-7e9989679f14 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:26.112646 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 628/1275] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:00:26 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:00:26.112646 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c5e2e07-a652-4a05-803e-4f1d7eb05265 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:26.118344 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-a7b0b7bc-1392-430a-9b86-7e9989679f14 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:00:26.120104 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-a7b0b7bc-1392-430a-9b86-7e9989679f14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:26.120362 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c5e2e07-a652-4a05-803e-4f1d7eb05265 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:26.124837 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d9ba36f-1ba0-4e74-b4dd-57fafa1b84a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:26.125449 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d9ba36f-1ba0-4e74-b4dd-57fafa1b84a5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:26.125724 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d9ba36f-1ba0-4e74-b4dd-57fafa1b84a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:26.125990 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d9ba36f-1ba0-4e74-b4dd-57fafa1b84a5 admin admin] PATH_INFO: `/v3/projects/16dda493652d4c87b075ee8cef7a8a95/users/8fbb2f982ece43a289217e5a6c16f3ab/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:26.127561 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 648/1276] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:00:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:00:26.128296 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7b0b7bc-1392-430a-9b86-7e9989679f14 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:26.138519 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7b0b7bc-1392-430a-9b86-7e9989679f14 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:26.143266 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81ce09f5-d10e-4fba-ba95-4f713211a5b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:26.143919 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ce09f5-d10e-4fba-ba95-4f713211a5b8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:26.144193 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ce09f5-d10e-4fba-ba95-4f713211a5b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:26.144462 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ce09f5-d10e-4fba-ba95-4f713211a5b8 admin admin] PATH_INFO: `/v3/projects/a393e550e71e4fa8a0ff9edd71382009/users/65f1034ab66f40dcb04da2872cfd64ef/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:26.144838 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d9ba36f-1ba0-4e74-b4dd-57fafa1b84a5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=16dda493652d4c87b075ee8cef7a8a95, user_id=8fbb2f982ece43a289217e5a6c16f3ab, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:26.147558 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d9ba36f-1ba0-4e74-b4dd-57fafa1b84a5 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:26.160178 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81ce09f5-d10e-4fba-ba95-4f713211a5b8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a393e550e71e4fa8a0ff9edd71382009, user_id=65f1034ab66f40dcb04da2872cfd64ef, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:26.160876 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 629/1277] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:00:26 2022] PUT /identity/v3/projects/16dda493652d4c87b075ee8cef7a8a95/users/8fbb2f982ece43a289217e5a6c16f3ab/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:00:26.161218 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: WARNING py.warnings [None req-81ce09f5-d10e-4fba-ba95-4f713211a5b8 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 20 09:00:26.161218 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: warnings.warn(msg) Oct 20 09:00:26.161218 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: Oct 20 09:00:26.161722 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d9ba36f-1ba0-4e74-b4dd-57fafa1b84a5 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:26.163224 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81ce09f5-d10e-4fba-ba95-4f713211a5b8 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:26.184021 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 649/1278] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:00:26 2022] PUT /identity/v3/projects/a393e550e71e4fa8a0ff9edd71382009/users/65f1034ab66f40dcb04da2872cfd64ef/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:00:26.185283 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5876aa1-d8ee-403f-a081-d33d41afd335 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:26.185568 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5876aa1-d8ee-403f-a081-d33d41afd335 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:26.185839 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5876aa1-d8ee-403f-a081-d33d41afd335 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:26.205151 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-b5876aa1-d8ee-403f-a081-d33d41afd335 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:00:26.206250 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-b5876aa1-d8ee-403f-a081-d33d41afd335 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:26.251611 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d9ba36f-1ba0-4e74-b4dd-57fafa1b84a5 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:26.267560 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 650/1279] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:00:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:00:26.268471 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5876aa1-d8ee-403f-a081-d33d41afd335 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:26.314695 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1727248-8e3a-4165-b238-e688105f65f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:26.315353 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1727248-8e3a-4165-b238-e688105f65f5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:26.315626 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1727248-8e3a-4165-b238-e688105f65f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:26.315894 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1727248-8e3a-4165-b238-e688105f65f5 admin admin] PATH_INFO: `/v3/projects/e5b0517639bf434cb67701de6f65ac14/users/5d07f82a43b948fca05cf8ce736a3715/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:26.323403 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5876aa1-d8ee-403f-a081-d33d41afd335 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:26.327799 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89f0e937-ad4e-4b4e-9b43-f9696a180546 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:26.328402 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f0e937-ad4e-4b4e-9b43-f9696a180546 service neutron] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:26.328680 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f0e937-ad4e-4b4e-9b43-f9696a180546 service neutron] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:26.328952 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f0e937-ad4e-4b4e-9b43-f9696a180546 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:26.329884 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1727248-8e3a-4165-b238-e688105f65f5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e5b0517639bf434cb67701de6f65ac14, user_id=5d07f82a43b948fca05cf8ce736a3715, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:26.332574 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1727248-8e3a-4165-b238-e688105f65f5 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:26.338177 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89f0e937-ad4e-4b4e-9b43-f9696a180546 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:26.339617 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89f0e937-ad4e-4b4e-9b43-f9696a180546 service neutron] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:26.346181 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 630/1280] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:00:26 2022] PUT /identity/v3/projects/e5b0517639bf434cb67701de6f65ac14/users/5d07f82a43b948fca05cf8ce736a3715/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:00:26.347183 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1727248-8e3a-4165-b238-e688105f65f5 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:26.354413 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 651/1281] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:00:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:00:26.355346 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89f0e937-ad4e-4b4e-9b43-f9696a180546 service neutron] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:26.365133 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89f0e937-ad4e-4b4e-9b43-f9696a180546 service neutron] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:26.392871 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:00:26.402974 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:00:26.434881 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f15de52b-2a92-41ab-8acd-50e60022a998 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:26.435552 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f15de52b-2a92-41ab-8acd-50e60022a998 admin admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:26.435831 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f15de52b-2a92-41ab-8acd-50e60022a998 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:26.436103 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f15de52b-2a92-41ab-8acd-50e60022a998 admin admin] PATH_INFO: `/v3/projects/cb4e66b2ef2b4d79b6b18a0e0c801719/users/c8b47ebff9924b218344fb930f083ce1/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:26.436544 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1727248-8e3a-4165-b238-e688105f65f5 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:26.443152 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9337de51-7691-4bd2-b738-0e69d32101d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:26.443969 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9337de51-7691-4bd2-b738-0e69d32101d8 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:26.444249 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9337de51-7691-4bd2-b738-0e69d32101d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:26.444555 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9337de51-7691-4bd2-b738-0e69d32101d8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:26.445198 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9337de51-7691-4bd2-b738-0e69d32101d8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:26.447164 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9337de51-7691-4bd2-b738-0e69d32101d8 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:26.457544 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f15de52b-2a92-41ab-8acd-50e60022a998 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cb4e66b2ef2b4d79b6b18a0e0c801719, user_id=c8b47ebff9924b218344fb930f083ce1, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:26.462794 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 631/1282] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:00:26 2022] GET /identity/v3/roles => generated 1420 bytes in 113 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:00:26.462984 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f15de52b-2a92-41ab-8acd-50e60022a998 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:26.463094 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9337de51-7691-4bd2-b738-0e69d32101d8 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:26.476499 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 652/1283] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:00:26 2022] PUT /identity/v3/projects/cb4e66b2ef2b4d79b6b18a0e0c801719/users/c8b47ebff9924b218344fb930f083ce1/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:00:26.478433 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47320c63-6234-48d4-a5b0-202f8494fd47 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:26.479037 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47320c63-6234-48d4-a5b0-202f8494fd47 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:26.479333 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47320c63-6234-48d4-a5b0-202f8494fd47 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:26.512258 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-47320c63-6234-48d4-a5b0-202f8494fd47 None None] MFA Rules not processed for user `8fbb2f982ece43a289217e5a6c16f3ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:00:26.513478 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-47320c63-6234-48d4-a5b0-202f8494fd47 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:26.543079 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9337de51-7691-4bd2-b738-0e69d32101d8 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:26.599510 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 653/1284] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:00:26 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:00:26.600851 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b77b2764-0d57-4750-95c9-b4b87070c657 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:26.601141 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b77b2764-0d57-4750-95c9-b4b87070c657 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:26.601395 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b77b2764-0d57-4750-95c9-b4b87070c657 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:26.604451 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd6b08f6-c37e-48ba-9065-31f34285d6e4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:26.605253 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6b08f6-c37e-48ba-9065-31f34285d6e4 service neutron] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:26.605533 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6b08f6-c37e-48ba-9065-31f34285d6e4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:26.605801 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6b08f6-c37e-48ba-9065-31f34285d6e4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:26.632963 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-b77b2764-0d57-4750-95c9-b4b87070c657 None None] MFA Rules not processed for user `65f1034ab66f40dcb04da2872cfd64ef`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:00:26.634206 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-b77b2764-0d57-4750-95c9-b4b87070c657 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:26.709267 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd6b08f6-c37e-48ba-9065-31f34285d6e4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:26.710910 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd6b08f6-c37e-48ba-9065-31f34285d6e4 service neutron] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:26.719080 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 654/1285] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:00:26 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:00:26.720278 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b77b2764-0d57-4750-95c9-b4b87070c657 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:26.733005 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 632/1286] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:00:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 272 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:00:26.733439 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b77b2764-0d57-4750-95c9-b4b87070c657 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:26.734392 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85915b4c-2ae1-4dc1-836f-f45884f76245 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:26.734796 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85915b4c-2ae1-4dc1-836f-f45884f76245 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:26.735072 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85915b4c-2ae1-4dc1-836f-f45884f76245 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:26.737326 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b995b4-e0a4-45ba-8db6-8c58fd4f3e57 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:26.738332 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b995b4-e0a4-45ba-8db6-8c58fd4f3e57 service neutron] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:26.738652 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b995b4-e0a4-45ba-8db6-8c58fd4f3e57 service neutron] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:26.739039 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b995b4-e0a4-45ba-8db6-8c58fd4f3e57 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:26.749962 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05b995b4-e0a4-45ba-8db6-8c58fd4f3e57 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:26.751618 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05b995b4-e0a4-45ba-8db6-8c58fd4f3e57 service neutron] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:26.765977 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 655/1287] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:00:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:00:26.767433 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b995b4-e0a4-45ba-8db6-8c58fd4f3e57 service neutron] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:26.773703 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-85915b4c-2ae1-4dc1-836f-f45884f76245 None None] MFA Rules not processed for user `5d07f82a43b948fca05cf8ce736a3715`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:00:26.776228 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-85915b4c-2ae1-4dc1-836f-f45884f76245 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:26.779094 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b995b4-e0a4-45ba-8db6-8c58fd4f3e57 service neutron] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:26.784674 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd3e5c9d-f477-4b25-aee1-b84d7a6025c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:26.785340 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd3e5c9d-f477-4b25-aee1-b84d7a6025c3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:26.785676 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd3e5c9d-f477-4b25-aee1-b84d7a6025c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:26.785965 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd3e5c9d-f477-4b25-aee1-b84d7a6025c3 admin admin] PATH_INFO: `/v3/projects/fcf51428b6bd4c428e1e01aadc6abe48/users/8002ab26f6d64822aaa65aa730781b76/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:26.791336 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:00:26.803829 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd3e5c9d-f477-4b25-aee1-b84d7a6025c3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fcf51428b6bd4c428e1e01aadc6abe48, user_id=8002ab26f6d64822aaa65aa730781b76, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:26.806880 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd3e5c9d-f477-4b25-aee1-b84d7a6025c3 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:26.839206 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 656/1288] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:00:26 2022] PUT /identity/v3/projects/fcf51428b6bd4c428e1e01aadc6abe48/users/8002ab26f6d64822aaa65aa730781b76/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 68 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:00:26.841211 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d11dda-35d8-4f20-9e7a-3d28720db9bf None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:26.841515 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d11dda-35d8-4f20-9e7a-3d28720db9bf None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:26.841824 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d11dda-35d8-4f20-9e7a-3d28720db9bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:26.849970 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:00:26.871769 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 633/1289] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:00:26 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:00:26.877337 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a1cc3e-d2ad-4b3f-849c-c3cf87cd4fd3 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:26.877725 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a1cc3e-d2ad-4b3f-849c-c3cf87cd4fd3 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:26.878055 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a1cc3e-d2ad-4b3f-849c-c3cf87cd4fd3 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:26.878267 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-79d11dda-35d8-4f20-9e7a-3d28720db9bf None None] MFA Rules not processed for user `c8b47ebff9924b218344fb930f083ce1`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:00:26.879561 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-79d11dda-35d8-4f20-9e7a-3d28720db9bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:26.879694 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 634/1290] 10.222.0.36 () {54 vars in 961 bytes} [Thu Oct 20 09:00:26 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 09:00:26.881073 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-938f69eb-d049-4f09-bd61-39611959c564 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:26.881405 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-938f69eb-d049-4f09-bd61-39611959c564 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:26.881679 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-938f69eb-d049-4f09-bd61-39611959c564 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:26.882912 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 635/1291] 10.222.0.36 () {54 vars in 961 bytes} [Thu Oct 20 09:00:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 09:00:26.884085 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d147de6-5b89-4c52-8a48-5ab06751a9c9 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:26.885016 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d147de6-5b89-4c52-8a48-5ab06751a9c9 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:26.885252 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d147de6-5b89-4c52-8a48-5ab06751a9c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:26.910229 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-0d147de6-5b89-4c52-8a48-5ab06751a9c9 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:00:26.911720 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-0d147de6-5b89-4c52-8a48-5ab06751a9c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:27.034631 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 636/1292] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:00:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:00:27.036199 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d147de6-5b89-4c52-8a48-5ab06751a9c9 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:27.049955 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 657/1293] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:00:26 2022] POST /identity/v3/auth/tokens => generated 3234 bytes in 207 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:00:27.050657 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d147de6-5b89-4c52-8a48-5ab06751a9c9 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:27.050953 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80fe587c-73ae-4150-a032-662d4da9ea87 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:27.051460 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80fe587c-73ae-4150-a032-662d4da9ea87 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:27.051791 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80fe587c-73ae-4150-a032-662d4da9ea87 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:27.057996 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d32ff59-cc3b-4ff7-9c14-f754e8b13efe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:27.059029 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d32ff59-cc3b-4ff7-9c14-f754e8b13efe admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:27.059284 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d32ff59-cc3b-4ff7-9c14-f754e8b13efe admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:27.059609 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d32ff59-cc3b-4ff7-9c14-f754e8b13efe admin admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:27.060325 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d32ff59-cc3b-4ff7-9c14-f754e8b13efe admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:27.063392 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d32ff59-cc3b-4ff7-9c14-f754e8b13efe admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:27.113995 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-80fe587c-73ae-4150-a032-662d4da9ea87 None None] MFA Rules not processed for user `a94e868115ec49918eda10724263930c`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:00:27.116037 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-80fe587c-73ae-4150-a032-662d4da9ea87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:27.117675 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 637/1294] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:00:27 2022] POST /identity/v3/projects => generated 363 bytes in 82 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:00:27.119529 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a386b28-6c38-4148-bcb9-b064e56d3a2b None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:27.120367 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a386b28-6c38-4148-bcb9-b064e56d3a2b None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:27.120924 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a386b28-6c38-4148-bcb9-b064e56d3a2b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:27.151781 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-6a386b28-6c38-4148-bcb9-b064e56d3a2b None None] MFA Rules not processed for user `a94e868115ec49918eda10724263930c`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:00:27.154044 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-6a386b28-6c38-4148-bcb9-b064e56d3a2b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:27.231371 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 658/1295] 10.222.0.36 () {58 vars in 1058 bytes} [Thu Oct 20 09:00:27 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:00:27.245086 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80fe587c-73ae-4150-a032-662d4da9ea87 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:27.247927 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-80fe587c-73ae-4150-a032-662d4da9ea87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:27.304629 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 638/1296] 10.222.0.36 () {58 vars in 1058 bytes} [Thu Oct 20 09:00:27 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:00:27.305574 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a386b28-6c38-4148-bcb9-b064e56d3a2b None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:27.375860 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:00:27.391396 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80fe587c-73ae-4150-a032-662d4da9ea87 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:27.396360 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31bafd3f-fcf2-46bd-be1d-86d17c6e117a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:27.396902 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31bafd3f-fcf2-46bd-be1d-86d17c6e117a service neutron] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:27.397248 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31bafd3f-fcf2-46bd-be1d-86d17c6e117a service neutron] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:27.397523 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31bafd3f-fcf2-46bd-be1d-86d17c6e117a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:27.446393 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:00:27.450338 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a386b28-6c38-4148-bcb9-b064e56d3a2b None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:27.456692 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e53c72bd-53ad-4821-a44e-448175dd98a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:27.457407 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53c72bd-53ad-4821-a44e-448175dd98a8 admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:27.457760 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53c72bd-53ad-4821-a44e-448175dd98a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:27.458320 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53c72bd-53ad-4821-a44e-448175dd98a8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:27.459193 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e53c72bd-53ad-4821-a44e-448175dd98a8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:27.461580 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e53c72bd-53ad-4821-a44e-448175dd98a8 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:27.530327 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 639/1297] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:00:27 2022] POST /identity/v3/projects => generated 357 bytes in 225 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:00:27.530540 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31bafd3f-fcf2-46bd-be1d-86d17c6e117a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:27.531247 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e53c72bd-53ad-4821-a44e-448175dd98a8 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:27.531883 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31bafd3f-fcf2-46bd-be1d-86d17c6e117a service neutron] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:27.556840 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 659/1298] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:00:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:00:27.557831 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31bafd3f-fcf2-46bd-be1d-86d17c6e117a service neutron] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:27.588403 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31bafd3f-fcf2-46bd-be1d-86d17c6e117a service neutron] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:27.623558 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e53c72bd-53ad-4821-a44e-448175dd98a8 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:27.634041 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e2ec26-afc9-468d-89db-8becebe0c760 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:27.635259 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e2ec26-afc9-468d-89db-8becebe0c760 admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:27.636630 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e2ec26-afc9-468d-89db-8becebe0c760 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:27.637813 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e2ec26-afc9-468d-89db-8becebe0c760 admin admin] PATH_INFO: `/v3/users` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:27.638522 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2e2ec26-afc9-468d-89db-8becebe0c760 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:27.645105 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2e2ec26-afc9-468d-89db-8becebe0c760 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:27.691444 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 640/1299] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:00:27 2022] POST /identity/v3/users => generated 288 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:00:27.692817 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e2ec26-afc9-468d-89db-8becebe0c760 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:27.730882 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba29ca0-a35e-4770-9322-b30d8803c71f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a94e868115ec49918eda10724263930c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:27.731468 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba29ca0-a35e-4770-9322-b30d8803c71f service nova] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:27.731752 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba29ca0-a35e-4770-9322-b30d8803c71f service nova] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:27.732025 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba29ca0-a35e-4770-9322-b30d8803c71f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:27.814099 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e2ec26-afc9-468d-89db-8becebe0c760 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:27.823191 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65bf8478-6f9c-436b-8baf-3f9f892ed55a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a94e868115ec49918eda10724263930c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:27.823828 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65bf8478-6f9c-436b-8baf-3f9f892ed55a service nova] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:27.824131 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65bf8478-6f9c-436b-8baf-3f9f892ed55a service nova] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:27.824378 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65bf8478-6f9c-436b-8baf-3f9f892ed55a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:27.871342 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ba29ca0-a35e-4770-9322-b30d8803c71f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:27.873926 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ba29ca0-a35e-4770-9322-b30d8803c71f service nova] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:27.891957 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:00:27.900178 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 660/1300] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:00:27 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 343 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:00:27.900906 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba29ca0-a35e-4770-9322-b30d8803c71f service nova] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:27.929045 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba29ca0-a35e-4770-9322-b30d8803c71f service nova] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:27.937201 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65bf8478-6f9c-436b-8baf-3f9f892ed55a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:27.939690 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65bf8478-6f9c-436b-8baf-3f9f892ed55a service nova] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:27.943288 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eabdcc1d-a375-460e-84dd-075b614125d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:27.943910 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eabdcc1d-a375-460e-84dd-075b614125d5 admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:27.944191 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eabdcc1d-a375-460e-84dd-075b614125d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:27.944466 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eabdcc1d-a375-460e-84dd-075b614125d5 admin admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:27.945102 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eabdcc1d-a375-460e-84dd-075b614125d5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:27.947394 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eabdcc1d-a375-460e-84dd-075b614125d5 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:27.965208 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 641/1301] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:00:27 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 273 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:00:27.967363 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65bf8478-6f9c-436b-8baf-3f9f892ed55a service nova] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:27.980306 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 661/1302] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:00:27 2022] POST /identity/v3/users => generated 285 bytes in 80 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:00:27.982980 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eabdcc1d-a375-460e-84dd-075b614125d5 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:27.989091 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65bf8478-6f9c-436b-8baf-3f9f892ed55a service nova] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:28.000707 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eabdcc1d-a375-460e-84dd-075b614125d5 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:28.009730 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be30fcf7-b397-40a9-87af-457b8aa26470 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:28.010367 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be30fcf7-b397-40a9-87af-457b8aa26470 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:28.010726 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be30fcf7-b397-40a9-87af-457b8aa26470 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:28.011078 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be30fcf7-b397-40a9-87af-457b8aa26470 admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:28.012908 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be30fcf7-b397-40a9-87af-457b8aa26470 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:28.014350 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be30fcf7-b397-40a9-87af-457b8aa26470 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:28.023219 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8b50bdc-6afb-4efe-8e8c-98964fcfdda8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:28.025872 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b50bdc-6afb-4efe-8e8c-98964fcfdda8 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:28.026230 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b50bdc-6afb-4efe-8e8c-98964fcfdda8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:28.027710 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b50bdc-6afb-4efe-8e8c-98964fcfdda8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:28.028461 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8b50bdc-6afb-4efe-8e8c-98964fcfdda8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:28.032374 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8b50bdc-6afb-4efe-8e8c-98964fcfdda8 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:28.056342 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 642/1303] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:00:27 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:00:28.057704 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be30fcf7-b397-40a9-87af-457b8aa26470 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:28.067191 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 662/1304] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:00:27 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:00:28.075890 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8b50bdc-6afb-4efe-8e8c-98964fcfdda8 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:28.084931 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be30fcf7-b397-40a9-87af-457b8aa26470 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:28.102083 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fd95ff3-dbf6-488f-b690-b81b998db6d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:28.102801 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd95ff3-dbf6-488f-b690-b81b998db6d6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:28.103670 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd95ff3-dbf6-488f-b690-b81b998db6d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:28.103929 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd95ff3-dbf6-488f-b690-b81b998db6d6 admin admin] PATH_INFO: `/v3/projects/838d5093ccd149ce882473932672a6bf/users/ea3852c085e04bd283fc75603f24a124/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:28.125057 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8b50bdc-6afb-4efe-8e8c-98964fcfdda8 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:28.142005 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fd95ff3-dbf6-488f-b690-b81b998db6d6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=838d5093ccd149ce882473932672a6bf, user_id=ea3852c085e04bd283fc75603f24a124, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:28.145140 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fd95ff3-dbf6-488f-b690-b81b998db6d6 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:28.156322 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a4843b5-02b3-450b-be29-2b24e81209cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:28.157918 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a4843b5-02b3-450b-be29-2b24e81209cc admin admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:28.158378 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a4843b5-02b3-450b-be29-2b24e81209cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:28.158690 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a4843b5-02b3-450b-be29-2b24e81209cc admin admin] PATH_INFO: `/v3/projects/49357129667f4f6581ec7b9dbb6c8fbc/users/9bdafc0e45434e5c954518259493df5d/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:28.179749 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a4843b5-02b3-450b-be29-2b24e81209cc admin admin] RBAC: Authorizing `identity:create_grant(project_id=49357129667f4f6581ec7b9dbb6c8fbc, user_id=9bdafc0e45434e5c954518259493df5d, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:28.183975 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 643/1305] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:00:28 2022] PUT /identity/v3/projects/838d5093ccd149ce882473932672a6bf/users/ea3852c085e04bd283fc75603f24a124/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 124 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:00:28.184250 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a4843b5-02b3-450b-be29-2b24e81209cc admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:28.185149 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fd95ff3-dbf6-488f-b690-b81b998db6d6 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:28.211687 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 663/1306] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:00:28 2022] PUT /identity/v3/projects/49357129667f4f6581ec7b9dbb6c8fbc/users/9bdafc0e45434e5c954518259493df5d/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:00:28.215857 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a4843b5-02b3-450b-be29-2b24e81209cc admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:28.234741 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:00:28.323374 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fd95ff3-dbf6-488f-b690-b81b998db6d6 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:28.366947 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a4843b5-02b3-450b-be29-2b24e81209cc admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:28.372242 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61fc7097-eeea-4324-a45c-09d9ac9f4246 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:28.372862 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61fc7097-eeea-4324-a45c-09d9ac9f4246 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:28.373140 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61fc7097-eeea-4324-a45c-09d9ac9f4246 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:28.373410 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61fc7097-eeea-4324-a45c-09d9ac9f4246 admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:28.374204 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61fc7097-eeea-4324-a45c-09d9ac9f4246 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:28.376074 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61fc7097-eeea-4324-a45c-09d9ac9f4246 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:28.391719 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 664/1307] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:00:28 2022] GET /identity/v3/roles => generated 1420 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:00:28.392608 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61fc7097-eeea-4324-a45c-09d9ac9f4246 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:28.404546 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61fc7097-eeea-4324-a45c-09d9ac9f4246 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:28.409695 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a8236a0-7da9-4f6d-bf0a-a249ed2fd082 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:28.410421 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8236a0-7da9-4f6d-bf0a-a249ed2fd082 admin admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:28.410806 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8236a0-7da9-4f6d-bf0a-a249ed2fd082 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:28.411088 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8236a0-7da9-4f6d-bf0a-a249ed2fd082 admin admin] PATH_INFO: `/v3/projects/49357129667f4f6581ec7b9dbb6c8fbc/users/9bdafc0e45434e5c954518259493df5d/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:28.413380 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a8236a0-7da9-4f6d-bf0a-a249ed2fd082 admin admin] RBAC: Authorizing `identity:create_grant(project_id=49357129667f4f6581ec7b9dbb6c8fbc, user_id=9bdafc0e45434e5c954518259493df5d, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:28.416433 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a8236a0-7da9-4f6d-bf0a-a249ed2fd082 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:28.428726 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 665/1308] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:00:28 2022] PUT /identity/v3/projects/49357129667f4f6581ec7b9dbb6c8fbc/users/9bdafc0e45434e5c954518259493df5d/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:00:28.439255 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733f8a6c-89f3-41dd-a0a8-ed8cbfd4b054 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:28.439563 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733f8a6c-89f3-41dd-a0a8-ed8cbfd4b054 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:28.439806 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733f8a6c-89f3-41dd-a0a8-ed8cbfd4b054 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:28.455647 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a83baf2a-7c81-4078-8c4c-5736f9523b9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:28.456330 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83baf2a-7c81-4078-8c4c-5736f9523b9e admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:28.456556 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83baf2a-7c81-4078-8c4c-5736f9523b9e admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:28.456849 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83baf2a-7c81-4078-8c4c-5736f9523b9e admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:28.457452 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a83baf2a-7c81-4078-8c4c-5736f9523b9e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:28.459103 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a83baf2a-7c81-4078-8c4c-5736f9523b9e admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:28.474298 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-733f8a6c-89f3-41dd-a0a8-ed8cbfd4b054 None None] MFA Rules not processed for user `9bdafc0e45434e5c954518259493df5d`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:00:28.475669 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-733f8a6c-89f3-41dd-a0a8-ed8cbfd4b054 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:28.480248 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 644/1309] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:00:28 2022] GET /identity/v3/roles => generated 1420 bytes in 296 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:00:28.484835 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a83baf2a-7c81-4078-8c4c-5736f9523b9e admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:28.505131 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a83baf2a-7c81-4078-8c4c-5736f9523b9e admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:28.597378 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 666/1310] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:00:28 2022] POST /identity/v3/auth/tokens => generated 3301 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:00:28.598829 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eacd328-e1f4-478e-bcc2-ff2af6c3fa31 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:28.599113 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eacd328-e1f4-478e-bcc2-ff2af6c3fa31 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:28.599381 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eacd328-e1f4-478e-bcc2-ff2af6c3fa31 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:28.601146 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 667/1311] 10.222.0.36 () {54 vars in 912 bytes} [Thu Oct 20 09:00:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 09:00:28.602324 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22131ae4-93ed-493e-bb1e-38407ddc1139 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:28.602632 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22131ae4-93ed-493e-bb1e-38407ddc1139 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:28.602940 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22131ae4-93ed-493e-bb1e-38407ddc1139 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:28.605912 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:00:28.608696 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 668/1312] 10.222.0.36 () {54 vars in 912 bytes} [Thu Oct 20 09:00:28 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 09:00:28.610582 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458a54c2-856b-43d2-898b-ab04c70bac2f None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:28.610827 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458a54c2-856b-43d2-898b-ab04c70bac2f None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:28.611852 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458a54c2-856b-43d2-898b-ab04c70bac2f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:28.628656 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9f97366-9c9d-43b3-87d6-740a37c65bf7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:28.629943 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9f97366-9c9d-43b3-87d6-740a37c65bf7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:28.630392 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9f97366-9c9d-43b3-87d6-740a37c65bf7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:28.630886 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9f97366-9c9d-43b3-87d6-740a37c65bf7 admin admin] PATH_INFO: `/v3/projects/838d5093ccd149ce882473932672a6bf/users/ea3852c085e04bd283fc75603f24a124/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:28.633474 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9f97366-9c9d-43b3-87d6-740a37c65bf7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=838d5093ccd149ce882473932672a6bf, user_id=ea3852c085e04bd283fc75603f24a124, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:28.633719 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-458a54c2-856b-43d2-898b-ab04c70bac2f None None] MFA Rules not processed for user `a94e868115ec49918eda10724263930c`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:00:28.635792 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-458a54c2-856b-43d2-898b-ab04c70bac2f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:28.639152 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9f97366-9c9d-43b3-87d6-740a37c65bf7 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:28.666122 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 645/1313] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:00:28 2022] PUT /identity/v3/projects/838d5093ccd149ce882473932672a6bf/users/ea3852c085e04bd283fc75603f24a124/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:00:28.669018 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb5fd0db-f6a8-4ee2-8d96-5a7dfcdf579a None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:28.669548 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb5fd0db-f6a8-4ee2-8d96-5a7dfcdf579a None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:28.669986 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb5fd0db-f6a8-4ee2-8d96-5a7dfcdf579a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:28.695536 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-bb5fd0db-f6a8-4ee2-8d96-5a7dfcdf579a None None] MFA Rules not processed for user `a94e868115ec49918eda10724263930c`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:00:28.700244 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-bb5fd0db-f6a8-4ee2-8d96-5a7dfcdf579a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:28.788979 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 669/1314] 10.222.0.36 () {58 vars in 1009 bytes} [Thu Oct 20 09:00:28 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:00:28.790099 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-458a54c2-856b-43d2-898b-ab04c70bac2f None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:28.801514 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:00:28.849817 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 646/1315] 10.222.0.36 () {58 vars in 1009 bytes} [Thu Oct 20 09:00:28 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:00:28.851503 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7f6c65-ff4c-4a71-b68b-b438c9e5ff01 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:28.851796 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7f6c65-ff4c-4a71-b68b-b438c9e5ff01 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:28.852530 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7f6c65-ff4c-4a71-b68b-b438c9e5ff01 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:28.902516 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-7a7f6c65-ff4c-4a71-b68b-b438c9e5ff01 None None] MFA Rules not processed for user `ea3852c085e04bd283fc75603f24a124`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:00:28.904369 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-7a7f6c65-ff4c-4a71-b68b-b438c9e5ff01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:28.908843 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-458a54c2-856b-43d2-898b-ab04c70bac2f None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:29.019916 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-840428e7-1742-4cff-914f-9c1f585030e5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:29.020767 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840428e7-1742-4cff-914f-9c1f585030e5 service cinder] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:29.021146 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840428e7-1742-4cff-914f-9c1f585030e5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:29.021564 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840428e7-1742-4cff-914f-9c1f585030e5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:29.057846 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 647/1316] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:00:28 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 207 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:00:29.059817 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a7f6c65-ff4c-4a71-b68b-b438c9e5ff01 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:29.072827 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a7f6c65-ff4c-4a71-b68b-b438c9e5ff01 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:29.077507 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b71ae795-116e-4e31-a51b-23cf98c670ca service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:29.078695 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b71ae795-116e-4e31-a51b-23cf98c670ca service cinder] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:29.079545 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b71ae795-116e-4e31-a51b-23cf98c670ca service cinder] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:29.080025 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b71ae795-116e-4e31-a51b-23cf98c670ca service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:29.100063 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b71ae795-116e-4e31-a51b-23cf98c670ca service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:29.102165 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b71ae795-116e-4e31-a51b-23cf98c670ca service cinder] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:29.120645 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 648/1317] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:00:29 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 62 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:00:29.156117 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-840428e7-1742-4cff-914f-9c1f585030e5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:29.158155 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-840428e7-1742-4cff-914f-9c1f585030e5 service cinder] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:29.187201 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 670/1318] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:00:28 2022] GET /identity/v3/auth/tokens => generated 3301 bytes in 397 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:00:29.257525 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b71ae795-116e-4e31-a51b-23cf98c670ca service cinder] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:29.278809 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b71ae795-116e-4e31-a51b-23cf98c670ca service cinder] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:29.285121 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26026470-17af-4c39-8039-27a7ab56a78e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:29.286267 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26026470-17af-4c39-8039-27a7ab56a78e service cinder] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:29.286771 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26026470-17af-4c39-8039-27a7ab56a78e service cinder] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:29.287200 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26026470-17af-4c39-8039-27a7ab56a78e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:29.307941 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26026470-17af-4c39-8039-27a7ab56a78e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:29.309574 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26026470-17af-4c39-8039-27a7ab56a78e service cinder] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:29.336634 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 649/1319] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:00:29 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 78 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:00:29.883213 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-840428e7-1742-4cff-914f-9c1f585030e5 service cinder] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:29.898973 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-840428e7-1742-4cff-914f-9c1f585030e5 service cinder] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:29.980902 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e263d75-bed5-4a33-a060-875869c06db6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:29.980902 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e263d75-bed5-4a33-a060-875869c06db6 service cinder] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:29.980902 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e263d75-bed5-4a33-a060-875869c06db6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:29.980902 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e263d75-bed5-4a33-a060-875869c06db6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:29.981546 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e263d75-bed5-4a33-a060-875869c06db6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:29.983283 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e263d75-bed5-4a33-a060-875869c06db6 service cinder] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:30.027173 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 671/1320] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:00:29 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 144 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:00:30.352473 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26026470-17af-4c39-8039-27a7ab56a78e service cinder] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:30.373479 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26026470-17af-4c39-8039-27a7ab56a78e service cinder] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:30.392258 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-364faaa8-ebca-4665-b3f1-9718e54606a4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:30.393361 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-364faaa8-ebca-4665-b3f1-9718e54606a4 service cinder] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:30.393932 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-364faaa8-ebca-4665-b3f1-9718e54606a4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:30.394222 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-364faaa8-ebca-4665-b3f1-9718e54606a4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:30.499658 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:00:30.554392 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-364faaa8-ebca-4665-b3f1-9718e54606a4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:30.556850 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-364faaa8-ebca-4665-b3f1-9718e54606a4 service cinder] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:30.570203 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 650/1321] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:00:30 2022] GET /identity/v3/auth/tokens => generated 3234 bytes in 218 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:00:30.631312 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e263d75-bed5-4a33-a060-875869c06db6 service cinder] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:30.652424 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e263d75-bed5-4a33-a060-875869c06db6 service cinder] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:30.661780 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-487c8203-15e6-44b4-a85f-56d316c08840 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:30.662458 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487c8203-15e6-44b4-a85f-56d316c08840 service cinder] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:30.662764 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487c8203-15e6-44b4-a85f-56d316c08840 service cinder] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:30.663924 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487c8203-15e6-44b4-a85f-56d316c08840 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:30.770249 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:00:30.773974 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-487c8203-15e6-44b4-a85f-56d316c08840 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:30.776790 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-487c8203-15e6-44b4-a85f-56d316c08840 service cinder] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:30.793942 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 672/1322] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:00:30 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 163 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:00:33.320348 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-364faaa8-ebca-4665-b3f1-9718e54606a4 service cinder] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:33.517269 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-364faaa8-ebca-4665-b3f1-9718e54606a4 service cinder] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:33.547691 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e472f04-affd-450b-96e8-3da698e70225 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:33.548501 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e472f04-affd-450b-96e8-3da698e70225 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:33.548980 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e472f04-affd-450b-96e8-3da698e70225 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:33.549756 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e472f04-affd-450b-96e8-3da698e70225 admin admin] PATH_INFO: `/v3/users/65f1034ab66f40dcb04da2872cfd64ef` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:33.552296 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e472f04-affd-450b-96e8-3da698e70225 admin admin] RBAC: Authorizing `identity:delete_user(user_id=65f1034ab66f40dcb04da2872cfd64ef)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:33.562977 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e472f04-affd-450b-96e8-3da698e70225 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:33.685381 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-6e472f04-affd-450b-96e8-3da698e70225 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '65f1034ab66f40dcb04da2872cfd64ef'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:33.706420 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-6e472f04-affd-450b-96e8-3da698e70225 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '65f1034ab66f40dcb04da2872cfd64ef'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:33.758585 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-6e472f04-affd-450b-96e8-3da698e70225 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '65f1034ab66f40dcb04da2872cfd64ef'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:33.770810 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-6e472f04-affd-450b-96e8-3da698e70225 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '65f1034ab66f40dcb04da2872cfd64ef'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:33.779363 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-6e472f04-affd-450b-96e8-3da698e70225 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '65f1034ab66f40dcb04da2872cfd64ef'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:33.805212 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 651/1323] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:00:33 2022] DELETE /identity/v3/users/65f1034ab66f40dcb04da2872cfd64ef => generated 0 bytes in 483 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:00:33.808177 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077af76e-23a0-4991-9424-2cac1e5decd7 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:33.809879 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077af76e-23a0-4991-9424-2cac1e5decd7 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:33.810330 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077af76e-23a0-4991-9424-2cac1e5decd7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:33.855279 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-077af76e-23a0-4991-9424-2cac1e5decd7 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:00:33.857437 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-077af76e-23a0-4991-9424-2cac1e5decd7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:33.977973 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 673/1324] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:00:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:00:33.999250 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e472f04-affd-450b-96e8-3da698e70225 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:34.001926 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-6e472f04-affd-450b-96e8-3da698e70225 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:34.141321 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:00:34.149846 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e472f04-affd-450b-96e8-3da698e70225 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:34.160580 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3a61366-2ba9-4a6f-9b28-575f4936c674 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:34.161691 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a61366-2ba9-4a6f-9b28-575f4936c674 service neutron] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:34.161974 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a61366-2ba9-4a6f-9b28-575f4936c674 service neutron] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:34.162352 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a61366-2ba9-4a6f-9b28-575f4936c674 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:34.189369 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3a61366-2ba9-4a6f-9b28-575f4936c674 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:34.191432 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3a61366-2ba9-4a6f-9b28-575f4936c674 service neutron] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:34.207438 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 652/1325] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:00:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 209 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 20 09:00:36.396695 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-077af76e-23a0-4991-9424-2cac1e5decd7 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:36.398238 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-077af76e-23a0-4991-9424-2cac1e5decd7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:36.421910 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-077af76e-23a0-4991-9424-2cac1e5decd7 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:36.435344 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7faff2e9-1445-47f6-9494-742d4f801a65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:36.438584 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7faff2e9-1445-47f6-9494-742d4f801a65 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:36.438937 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7faff2e9-1445-47f6-9494-742d4f801a65 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:36.439295 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7faff2e9-1445-47f6-9494-742d4f801a65 admin admin] PATH_INFO: `/v3/projects/a393e550e71e4fa8a0ff9edd71382009` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:36.443491 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7faff2e9-1445-47f6-9494-742d4f801a65 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a393e550e71e4fa8a0ff9edd71382009)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:36.447625 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7faff2e9-1445-47f6-9494-742d4f801a65 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:36.567865 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-7faff2e9-1445-47f6-9494-742d4f801a65 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a393e550e71e4fa8a0ff9edd71382009', 'request_id': 'req-7faff2e9-1445-47f6-9494-742d4f801a65'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:36.584639 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-7faff2e9-1445-47f6-9494-742d4f801a65 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a393e550e71e4fa8a0ff9edd71382009', 'request_id': 'req-7faff2e9-1445-47f6-9494-742d4f801a65'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:36.604320 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-7faff2e9-1445-47f6-9494-742d4f801a65 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a393e550e71e4fa8a0ff9edd71382009', 'request_id': 'req-7faff2e9-1445-47f6-9494-742d4f801a65'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:36.637180 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-7faff2e9-1445-47f6-9494-742d4f801a65 admin admin] The token cache is being invalidate because project a393e550e71e4fa8a0ff9edd71382009 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81562) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:00:36.637528 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-7faff2e9-1445-47f6-9494-742d4f801a65 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:36.639373 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-7faff2e9-1445-47f6-9494-742d4f801a65 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:36.645625 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 674/1326] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:00:36 2022] DELETE /identity/v3/projects/a393e550e71e4fa8a0ff9edd71382009 => generated 0 bytes in 249 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:00:36.650795 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea851c2-07b6-4afa-992d-1b48a4a0fa6b None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:36.651330 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea851c2-07b6-4afa-992d-1b48a4a0fa6b None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:36.651428 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea851c2-07b6-4afa-992d-1b48a4a0fa6b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:36.718078 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-2ea851c2-07b6-4afa-992d-1b48a4a0fa6b None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:00:36.720877 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-2ea851c2-07b6-4afa-992d-1b48a4a0fa6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:36.924749 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 653/1327] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:00:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 276 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:00:36.933169 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee615bd-6536-4b02-b1a4-17a020ad411e None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:36.933868 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee615bd-6536-4b02-b1a4-17a020ad411e None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:36.934238 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee615bd-6536-4b02-b1a4-17a020ad411e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:36.982210 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-eee615bd-6536-4b02-b1a4-17a020ad411e None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:00:37.000020 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-eee615bd-6536-4b02-b1a4-17a020ad411e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:37.012422 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 675/1328] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:00:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:00:37.018888 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea851c2-07b6-4afa-992d-1b48a4a0fa6b None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:37.035981 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea851c2-07b6-4afa-992d-1b48a4a0fa6b None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:37.054014 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8687aa24-1229-45c0-a081-7731752be87c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:37.057193 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8687aa24-1229-45c0-a081-7731752be87c admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:37.057905 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8687aa24-1229-45c0-a081-7731752be87c admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:37.058189 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8687aa24-1229-45c0-a081-7731752be87c admin admin] PATH_INFO: `/v3/domains` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:37.059245 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8687aa24-1229-45c0-a081-7731752be87c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 09:00:37.059793 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8687aa24-1229-45c0-a081-7731752be87c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:37.063588 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8687aa24-1229-45c0-a081-7731752be87c admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:37.089645 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 654/1329] 10.222.0.36 () {58 vars in 1153 bytes} [Thu Oct 20 09:00:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 74 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 09:00:37.093960 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eee615bd-6536-4b02-b1a4-17a020ad411e None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:37.106405 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eee615bd-6536-4b02-b1a4-17a020ad411e None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:37.124138 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd781077-337c-4fc9-9823-7c9da76e8acd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:37.124692 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd781077-337c-4fc9-9823-7c9da76e8acd admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:37.124970 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd781077-337c-4fc9-9823-7c9da76e8acd admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:37.125238 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd781077-337c-4fc9-9823-7c9da76e8acd admin admin] PATH_INFO: `/v3/projects` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:37.125880 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd781077-337c-4fc9-9823-7c9da76e8acd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:37.128093 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd781077-337c-4fc9-9823-7c9da76e8acd admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:37.224920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8687aa24-1229-45c0-a081-7731752be87c admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:37.226877 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 676/1330] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:00:37 2022] POST /identity/v3/projects => generated 379 bytes in 133 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 20 09:00:37.240264 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8687aa24-1229-45c0-a081-7731752be87c admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:37.437317 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92142c8-ec1f-4801-b0d3-f3aa9683d5c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:37.437985 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92142c8-ec1f-4801-b0d3-f3aa9683d5c5 admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:37.438269 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92142c8-ec1f-4801-b0d3-f3aa9683d5c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:37.440463 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92142c8-ec1f-4801-b0d3-f3aa9683d5c5 admin admin] PATH_INFO: `/v3/users` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:37.441175 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b92142c8-ec1f-4801-b0d3-f3aa9683d5c5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:37.443332 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b92142c8-ec1f-4801-b0d3-f3aa9683d5c5 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:37.505863 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 655/1331] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:00:37 2022] POST /identity/v3/users => generated 290 bytes in 281 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:00:37.510079 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd781077-337c-4fc9-9823-7c9da76e8acd admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:37.527211 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd781077-337c-4fc9-9823-7c9da76e8acd admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:37.535811 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98a94439-fb3a-4350-a2af-9db1810dbb63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:37.538786 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a94439-fb3a-4350-a2af-9db1810dbb63 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:37.539477 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a94439-fb3a-4350-a2af-9db1810dbb63 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:37.540762 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a94439-fb3a-4350-a2af-9db1810dbb63 admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:37.542433 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98a94439-fb3a-4350-a2af-9db1810dbb63 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:37.544839 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98a94439-fb3a-4350-a2af-9db1810dbb63 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:37.565514 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 677/1332] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:00:37 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:00:37.569337 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92142c8-ec1f-4801-b0d3-f3aa9683d5c5 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:37.586050 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92142c8-ec1f-4801-b0d3-f3aa9683d5c5 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:37.593907 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80419a16-4b18-481d-938f-f0774d467a54 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:37.595093 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80419a16-4b18-481d-938f-f0774d467a54 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:37.595381 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80419a16-4b18-481d-938f-f0774d467a54 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:37.596981 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80419a16-4b18-481d-938f-f0774d467a54 admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:37.598902 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80419a16-4b18-481d-938f-f0774d467a54 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:37.607453 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80419a16-4b18-481d-938f-f0774d467a54 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:37.646059 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 656/1333] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:00:37 2022] GET /identity/v3/roles => generated 1420 bytes in 77 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:00:37.651629 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98a94439-fb3a-4350-a2af-9db1810dbb63 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:37.675122 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98a94439-fb3a-4350-a2af-9db1810dbb63 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:37.684380 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f54357e0-b160-4c18-aa3a-9dedbdcda9c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:37.689451 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80419a16-4b18-481d-938f-f0774d467a54 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:37.691347 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-80419a16-4b18-481d-938f-f0774d467a54 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:37.697871 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54357e0-b160-4c18-aa3a-9dedbdcda9c1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:37.700622 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54357e0-b160-4c18-aa3a-9dedbdcda9c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:37.703246 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54357e0-b160-4c18-aa3a-9dedbdcda9c1 admin admin] PATH_INFO: `/v3/projects/ca0fdf3f133548f382e6072c8944538f/users/7db2cb676413491dade7fddb604cef46/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:37.734443 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80419a16-4b18-481d-938f-f0774d467a54 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:37.754366 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3a7d4db-bc55-450d-86af-b4fe21b2a0fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:37.755217 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a7d4db-bc55-450d-86af-b4fe21b2a0fe admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:37.755488 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a7d4db-bc55-450d-86af-b4fe21b2a0fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:37.755814 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a7d4db-bc55-450d-86af-b4fe21b2a0fe admin admin] PATH_INFO: `/v3/users/5d07f82a43b948fca05cf8ce736a3715` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:37.757860 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3a7d4db-bc55-450d-86af-b4fe21b2a0fe admin admin] RBAC: Authorizing `identity:delete_user(user_id=5d07f82a43b948fca05cf8ce736a3715)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:37.758617 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f54357e0-b160-4c18-aa3a-9dedbdcda9c1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ca0fdf3f133548f382e6072c8944538f, user_id=7db2cb676413491dade7fddb604cef46, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:37.764099 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3a7d4db-bc55-450d-86af-b4fe21b2a0fe admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:37.766001 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f54357e0-b160-4c18-aa3a-9dedbdcda9c1 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:37.823983 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 678/1334] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:00:37 2022] PUT /identity/v3/projects/ca0fdf3f133548f382e6072c8944538f/users/7db2cb676413491dade7fddb604cef46/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:00:37.829897 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f125f08-e61e-4b66-9889-703944ab2b0a None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:37.830210 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f125f08-e61e-4b66-9889-703944ab2b0a None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:37.830507 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f125f08-e61e-4b66-9889-703944ab2b0a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:37.927287 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-5f125f08-e61e-4b66-9889-703944ab2b0a None None] MFA Rules not processed for user `7db2cb676413491dade7fddb604cef46`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:00:37.929159 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-5f125f08-e61e-4b66-9889-703944ab2b0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:37.934426 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:00:37.984238 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-e3a7d4db-bc55-450d-86af-b4fe21b2a0fe admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5d07f82a43b948fca05cf8ce736a3715'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:38.004714 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=88,keystone:INSERT=4,keystone:DELETE=8 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:00:38.021573 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-e3a7d4db-bc55-450d-86af-b4fe21b2a0fe admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5d07f82a43b948fca05cf8ce736a3715'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:38.054833 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-e3a7d4db-bc55-450d-86af-b4fe21b2a0fe admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5d07f82a43b948fca05cf8ce736a3715'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:38.072005 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-e3a7d4db-bc55-450d-86af-b4fe21b2a0fe admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5d07f82a43b948fca05cf8ce736a3715'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:38.073206 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-e3a7d4db-bc55-450d-86af-b4fe21b2a0fe admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5d07f82a43b948fca05cf8ce736a3715'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:38.093606 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 657/1335] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:00:37 2022] DELETE /identity/v3/users/5d07f82a43b948fca05cf8ce736a3715 => generated 0 bytes in 405 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:00:38.098928 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff8c515-69ed-469c-8948-ce797f38a32c None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:38.099315 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff8c515-69ed-469c-8948-ce797f38a32c None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:38.099685 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff8c515-69ed-469c-8948-ce797f38a32c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:38.140595 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-9ff8c515-69ed-469c-8948-ce797f38a32c None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:00:38.142403 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-9ff8c515-69ed-469c-8948-ce797f38a32c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:38.153247 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 679/1336] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:00:37 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 328 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:00:38.158729 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f125f08-e61e-4b66-9889-703944ab2b0a None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:38.167812 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-5f125f08-e61e-4b66-9889-703944ab2b0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:38.283180 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 658/1337] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:00:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:00:38.329751 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ff8c515-69ed-469c-8948-ce797f38a32c None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:38.331201 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-9ff8c515-69ed-469c-8948-ce797f38a32c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:38.363519 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f125f08-e61e-4b66-9889-703944ab2b0a None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:38.382872 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b0cc44-89b0-453e-a37e-2b57e2d738fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:38.384002 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b0cc44-89b0-453e-a37e-2b57e2d738fb admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:38.384301 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b0cc44-89b0-453e-a37e-2b57e2d738fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:38.384590 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b0cc44-89b0-453e-a37e-2b57e2d738fb admin admin] PATH_INFO: `/v3/projects` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:38.385222 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3b0cc44-89b0-453e-a37e-2b57e2d738fb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:38.387319 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3b0cc44-89b0-453e-a37e-2b57e2d738fb admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:38.462308 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ff8c515-69ed-469c-8948-ce797f38a32c None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:38.466627 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8f0db22-3b0d-4160-a77b-8e469c7fe39a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:38.467369 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f0db22-3b0d-4160-a77b-8e469c7fe39a service neutron] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:38.468181 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f0db22-3b0d-4160-a77b-8e469c7fe39a service neutron] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:38.468181 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f0db22-3b0d-4160-a77b-8e469c7fe39a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:38.480905 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8f0db22-3b0d-4160-a77b-8e469c7fe39a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:38.483169 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8f0db22-3b0d-4160-a77b-8e469c7fe39a service neutron] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:38.485981 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 680/1338] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:00:38 2022] POST /identity/v3/projects => generated 381 bytes in 328 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:00:38.490169 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b0cc44-89b0-453e-a37e-2b57e2d738fb admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:38.500964 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 659/1339] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:00:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:00:38.651062 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b0cc44-89b0-453e-a37e-2b57e2d738fb admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:38.661350 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f64d27-63bc-42a7-8f7c-e21727b39a10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:38.661993 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f64d27-63bc-42a7-8f7c-e21727b39a10 admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:38.662268 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f64d27-63bc-42a7-8f7c-e21727b39a10 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:38.662564 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f64d27-63bc-42a7-8f7c-e21727b39a10 admin admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:38.663223 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20f64d27-63bc-42a7-8f7c-e21727b39a10 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:38.665736 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20f64d27-63bc-42a7-8f7c-e21727b39a10 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:38.730338 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 681/1340] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:00:38 2022] POST /identity/v3/users => generated 297 bytes in 240 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 20 09:00:38.730872 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8f0db22-3b0d-4160-a77b-8e469c7fe39a service neutron] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:38.745091 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8f0db22-3b0d-4160-a77b-8e469c7fe39a service neutron] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:38.757574 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-944cfd02-3618-49c8-b777-2cf5e4973df4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:38.758840 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-944cfd02-3618-49c8-b777-2cf5e4973df4 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:38.759256 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-944cfd02-3618-49c8-b777-2cf5e4973df4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:38.760320 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-944cfd02-3618-49c8-b777-2cf5e4973df4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:38.760616 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-944cfd02-3618-49c8-b777-2cf5e4973df4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:38.763122 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-944cfd02-3618-49c8-b777-2cf5e4973df4 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:38.780497 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f64d27-63bc-42a7-8f7c-e21727b39a10 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:38.793263 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 660/1341] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:00:38 2022] GET /identity/v3/roles => generated 1420 bytes in 61 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 20 09:00:38.794320 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f64d27-63bc-42a7-8f7c-e21727b39a10 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:38.799346 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03229610-c749-4afb-8c9c-07da95ba068a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:38.800339 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03229610-c749-4afb-8c9c-07da95ba068a admin admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:38.801029 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03229610-c749-4afb-8c9c-07da95ba068a admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:38.801418 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03229610-c749-4afb-8c9c-07da95ba068a admin admin] PATH_INFO: `/v3/projects/45f38f69246e473e88cb94e0cc264524/users/5969af5e5b8b4f849218478f86d3f631/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:38.834913 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03229610-c749-4afb-8c9c-07da95ba068a admin admin] RBAC: Authorizing `identity:create_grant(project_id=45f38f69246e473e88cb94e0cc264524, user_id=5969af5e5b8b4f849218478f86d3f631, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:38.843848 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03229610-c749-4afb-8c9c-07da95ba068a admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:38.851921 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:00:38.893973 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-944cfd02-3618-49c8-b777-2cf5e4973df4 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:38.904610 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 682/1342] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:00:38 2022] PUT /identity/v3/projects/45f38f69246e473e88cb94e0cc264524/users/5969af5e5b8b4f849218478f86d3f631/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:00:38.906853 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-944cfd02-3618-49c8-b777-2cf5e4973df4 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:39.017889 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eff63df-02bf-4652-9e76-06f00ee79e07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:39.018728 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eff63df-02bf-4652-9e76-06f00ee79e07 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:39.019123 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eff63df-02bf-4652-9e76-06f00ee79e07 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:39.019546 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eff63df-02bf-4652-9e76-06f00ee79e07 admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:39.020338 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eff63df-02bf-4652-9e76-06f00ee79e07 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:39.030311 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eff63df-02bf-4652-9e76-06f00ee79e07 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:39.034255 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:00:39.066071 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03229610-c749-4afb-8c9c-07da95ba068a admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:39.073402 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 661/1343] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:00:38 2022] GET /identity/v3/roles => generated 1420 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:00:39.081074 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03229610-c749-4afb-8c9c-07da95ba068a admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:39.094607 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03a1e438-55df-45a4-abcf-266e5ff21fbd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:39.095210 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a1e438-55df-45a4-abcf-266e5ff21fbd admin admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:39.100964 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a1e438-55df-45a4-abcf-266e5ff21fbd admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:39.101347 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a1e438-55df-45a4-abcf-266e5ff21fbd admin admin] PATH_INFO: `/v3/projects/45f38f69246e473e88cb94e0cc264524/users/5969af5e5b8b4f849218478f86d3f631/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:39.103719 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03a1e438-55df-45a4-abcf-266e5ff21fbd admin admin] RBAC: Authorizing `identity:create_grant(project_id=45f38f69246e473e88cb94e0cc264524, user_id=5969af5e5b8b4f849218478f86d3f631, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:39.109357 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03a1e438-55df-45a4-abcf-266e5ff21fbd admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:39.159570 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 683/1344] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:00:39 2022] PUT /identity/v3/projects/45f38f69246e473e88cb94e0cc264524/users/5969af5e5b8b4f849218478f86d3f631/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 87 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:00:39.163346 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d14e037-d071-415c-8308-d10bee2324e5 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:39.163698 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d14e037-d071-415c-8308-d10bee2324e5 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:39.163977 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d14e037-d071-415c-8308-d10bee2324e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:39.221252 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-3d14e037-d071-415c-8308-d10bee2324e5 None None] MFA Rules not processed for user `5969af5e5b8b4f849218478f86d3f631`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:00:39.223147 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-3d14e037-d071-415c-8308-d10bee2324e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:39.392220 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 662/1345] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:00:39 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 228 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:00:39.404971 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03a1e438-55df-45a4-abcf-266e5ff21fbd admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:39.406985 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-03a1e438-55df-45a4-abcf-266e5ff21fbd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:39.522339 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3bc1987-d675-47d6-a37a-d68a57ff56af None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:39.524438 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3bc1987-d675-47d6-a37a-d68a57ff56af None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:39.525001 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3bc1987-d675-47d6-a37a-d68a57ff56af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:39.613924 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-a3bc1987-d675-47d6-a37a-d68a57ff56af None None] MFA Rules not processed for user `c7c40a4592874732871eaabd04d8662c`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:00:39.619961 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-a3bc1987-d675-47d6-a37a-d68a57ff56af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:39.691792 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03a1e438-55df-45a4-abcf-266e5ff21fbd admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:39.709291 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-023f6d73-497a-442b-baf6-452fca1b61eb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:39.710441 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-023f6d73-497a-442b-baf6-452fca1b61eb service cinder] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:39.710973 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-023f6d73-497a-442b-baf6-452fca1b61eb service cinder] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:39.711460 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-023f6d73-497a-442b-baf6-452fca1b61eb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:39.750028 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-023f6d73-497a-442b-baf6-452fca1b61eb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:39.752868 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-023f6d73-497a-442b-baf6-452fca1b61eb service cinder] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:39.792743 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 684/1346] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:00:39 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 390 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:00:39.851129 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea7f085-1190-418b-b3f7-683ac8aae90c None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:39.851929 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea7f085-1190-418b-b3f7-683ac8aae90c None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:39.852280 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea7f085-1190-418b-b3f7-683ac8aae90c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:39.865690 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 663/1347] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:00:39 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 339 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:00:39.893159 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-bea7f085-1190-418b-b3f7-683ac8aae90c None None] MFA Rules not processed for user `c7c40a4592874732871eaabd04d8662c`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:00:39.900869 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-bea7f085-1190-418b-b3f7-683ac8aae90c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:39.946936 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 685/1348] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:00:39 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:00:39.959557 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3bc1987-d675-47d6-a37a-d68a57ff56af None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:39.961256 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-a3bc1987-d675-47d6-a37a-d68a57ff56af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:40.043680 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea7f085-1190-418b-b3f7-683ac8aae90c None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:40.060820 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea7f085-1190-418b-b3f7-683ac8aae90c None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:40.084744 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d81f295f-d5b9-42bd-a166-c0b3057cbe09 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:40.087652 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d81f295f-d5b9-42bd-a166-c0b3057cbe09 service cinder] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:40.088791 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d81f295f-d5b9-42bd-a166-c0b3057cbe09 service cinder] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:40.089276 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d81f295f-d5b9-42bd-a166-c0b3057cbe09 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:40.208252 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3bc1987-d675-47d6-a37a-d68a57ff56af None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:40.236564 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d00a879b-3a49-41c5-ad5c-c5b4c04a907d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a94e868115ec49918eda10724263930c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:40.237793 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d00a879b-3a49-41c5-ad5c-c5b4c04a907d service nova] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:40.238077 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d00a879b-3a49-41c5-ad5c-c5b4c04a907d service nova] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:40.239530 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d00a879b-3a49-41c5-ad5c-c5b4c04a907d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:40.260773 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d00a879b-3a49-41c5-ad5c-c5b4c04a907d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:40.271425 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d00a879b-3a49-41c5-ad5c-c5b4c04a907d service nova] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:40.276100 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:00:40.291076 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d81f295f-d5b9-42bd-a166-c0b3057cbe09 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:40.292346 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 664/1349] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:00:39 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 333 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:00:40.293223 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d81f295f-d5b9-42bd-a166-c0b3057cbe09 service cinder] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:40.322072 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 686/1350] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:00:40 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 279 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 20 09:00:40.559927 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f6c919-2d90-4f19-8ec9-4b47c9504488 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:40.560220 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f6c919-2d90-4f19-8ec9-4b47c9504488 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:40.560499 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f6c919-2d90-4f19-8ec9-4b47c9504488 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:40.662282 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-84f6c919-2d90-4f19-8ec9-4b47c9504488 None None] MFA Rules not processed for user `7f932695a5994c5790d14c210ea65326`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:00:40.663703 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-84f6c919-2d90-4f19-8ec9-4b47c9504488 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:40.829581 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 665/1351] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:00:40 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 271 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:00:40.840281 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c3d80dd-509a-464b-b1c3-ec2ee95a8440 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:40.841453 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c3d80dd-509a-464b-b1c3-ec2ee95a8440 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:40.841932 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c3d80dd-509a-464b-b1c3-ec2ee95a8440 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:40.942868 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-0c3d80dd-509a-464b-b1c3-ec2ee95a8440 None None] MFA Rules not processed for user `7f932695a5994c5790d14c210ea65326`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:00:40.944516 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-0c3d80dd-509a-464b-b1c3-ec2ee95a8440 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:40.987099 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 687/1352] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:00:40 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:00:41.112381 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84f6c919-2d90-4f19-8ec9-4b47c9504488 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:41.114040 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-84f6c919-2d90-4f19-8ec9-4b47c9504488 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:41.169174 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84f6c919-2d90-4f19-8ec9-4b47c9504488 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:41.190221 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb47e672-1b79-4dc0-a0c3-10f6ac73643e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a94e868115ec49918eda10724263930c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:41.191256 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb47e672-1b79-4dc0-a0c3-10f6ac73643e service nova] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:41.191715 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb47e672-1b79-4dc0-a0c3-10f6ac73643e service nova] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:41.192166 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb47e672-1b79-4dc0-a0c3-10f6ac73643e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:41.199408 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac30082b-dfd1-4985-b68d-2cb1ff9a68d3 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:41.199698 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac30082b-dfd1-4985-b68d-2cb1ff9a68d3 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:41.199979 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac30082b-dfd1-4985-b68d-2cb1ff9a68d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:41.245470 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb47e672-1b79-4dc0-a0c3-10f6ac73643e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:41.247327 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb47e672-1b79-4dc0-a0c3-10f6ac73643e service nova] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:41.268657 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 666/1353] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:00:41 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 157 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:00:41.269788 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb47e672-1b79-4dc0-a0c3-10f6ac73643e service nova] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:41.274860 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-ac30082b-dfd1-4985-b68d-2cb1ff9a68d3 None None] MFA Rules not processed for user `fd9733d1bb2b4520b70090542615ddf9`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:00:41.276087 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-ac30082b-dfd1-4985-b68d-2cb1ff9a68d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:41.278676 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-cb47e672-1b79-4dc0-a0c3-10f6ac73643e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:41.522830 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:00:41.541182 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 688/1354] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:00:41 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 343 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:00:41.563128 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb47e672-1b79-4dc0-a0c3-10f6ac73643e service nova] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:41.573449 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed93428-535c-45f8-b326-fa48d244ee43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:41.574123 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed93428-535c-45f8-b326-fa48d244ee43 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:41.574406 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed93428-535c-45f8-b326-fa48d244ee43 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:41.574715 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed93428-535c-45f8-b326-fa48d244ee43 admin admin] PATH_INFO: `/v3/projects/e5b0517639bf434cb67701de6f65ac14` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:41.579311 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bed93428-535c-45f8-b326-fa48d244ee43 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e5b0517639bf434cb67701de6f65ac14)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:41.580511 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: WARNING py.warnings [None req-bed93428-535c-45f8-b326-fa48d244ee43 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 20 09:00:41.580511 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: warnings.warn(msg) Oct 20 09:00:41.580511 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: Oct 20 09:00:41.581747 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bed93428-535c-45f8-b326-fa48d244ee43 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:41.728209 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-bed93428-535c-45f8-b326-fa48d244ee43 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e5b0517639bf434cb67701de6f65ac14', 'request_id': 'req-bed93428-535c-45f8-b326-fa48d244ee43'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:41.800667 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-bed93428-535c-45f8-b326-fa48d244ee43 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e5b0517639bf434cb67701de6f65ac14', 'request_id': 'req-bed93428-535c-45f8-b326-fa48d244ee43'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:41.819493 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-bed93428-535c-45f8-b326-fa48d244ee43 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e5b0517639bf434cb67701de6f65ac14', 'request_id': 'req-bed93428-535c-45f8-b326-fa48d244ee43'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:41.869901 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-bed93428-535c-45f8-b326-fa48d244ee43 admin admin] The token cache is being invalidate because project e5b0517639bf434cb67701de6f65ac14 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81561) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:00:41.870199 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-bed93428-535c-45f8-b326-fa48d244ee43 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:41.874174 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-bed93428-535c-45f8-b326-fa48d244ee43 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:41.876079 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 667/1355] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:00:41 2022] DELETE /identity/v3/projects/e5b0517639bf434cb67701de6f65ac14 => generated 0 bytes in 607 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:00:41.879965 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac30082b-dfd1-4985-b68d-2cb1ff9a68d3 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:41.905179 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac30082b-dfd1-4985-b68d-2cb1ff9a68d3 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:41.949940 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5901a7ff-49df-4bc0-9db6-fb5a3870f8ab None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:41.950425 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5901a7ff-49df-4bc0-9db6-fb5a3870f8ab None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:41.950916 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5901a7ff-49df-4bc0-9db6-fb5a3870f8ab None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:41.952923 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 668/1356] 10.222.0.36 () {54 vars in 912 bytes} [Thu Oct 20 09:00:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 20 09:00:41.958626 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f082e55b-d14f-4b8c-a485-1f5fae04881c None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:41.959187 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f082e55b-d14f-4b8c-a485-1f5fae04881c None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:41.959412 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f082e55b-d14f-4b8c-a485-1f5fae04881c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:41.986383 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:00:42.073922 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-f082e55b-d14f-4b8c-a485-1f5fae04881c None None] MFA Rules not processed for user `794ca389002d41748d7925216ed964a5`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:00:42.076100 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-f082e55b-d14f-4b8c-a485-1f5fae04881c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:42.089245 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56af5a89-2055-4cb9-9eee-bca280daf97d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:42.091493 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56af5a89-2055-4cb9-9eee-bca280daf97d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:42.092810 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56af5a89-2055-4cb9-9eee-bca280daf97d admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:42.094353 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56af5a89-2055-4cb9-9eee-bca280daf97d admin admin] PATH_INFO: `/v3/users/ea3852c085e04bd283fc75603f24a124` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:42.101181 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56af5a89-2055-4cb9-9eee-bca280daf97d admin admin] RBAC: Authorizing `identity:delete_user(user_id=ea3852c085e04bd283fc75603f24a124)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:42.103575 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: WARNING py.warnings [None req-56af5a89-2055-4cb9-9eee-bca280daf97d admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 20 09:00:42.103575 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: warnings.warn(msg) Oct 20 09:00:42.103575 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: Oct 20 09:00:42.106780 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56af5a89-2055-4cb9-9eee-bca280daf97d admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:42.227078 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 669/1357] 10.222.0.36 () {58 vars in 1009 bytes} [Thu Oct 20 09:00:41 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 269 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 20 09:00:42.312751 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-56af5a89-2055-4cb9-9eee-bca280daf97d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ea3852c085e04bd283fc75603f24a124'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:42.326175 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-56af5a89-2055-4cb9-9eee-bca280daf97d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ea3852c085e04bd283fc75603f24a124'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:42.338104 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c38022-f9fa-4d02-aed0-082b62fc54b2 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:42.339325 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c38022-f9fa-4d02-aed0-082b62fc54b2 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:42.339599 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c38022-f9fa-4d02-aed0-082b62fc54b2 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:42.342176 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 670/1358] 10.222.0.36 () {54 vars in 912 bytes} [Thu Oct 20 09:00:42 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 09:00:42.349174 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e446c267-68c8-4db7-9286-19a45eb14065 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:42.349550 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e446c267-68c8-4db7-9286-19a45eb14065 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:42.349825 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e446c267-68c8-4db7-9286-19a45eb14065 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:42.372954 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-56af5a89-2055-4cb9-9eee-bca280daf97d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ea3852c085e04bd283fc75603f24a124'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:42.408545 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-56af5a89-2055-4cb9-9eee-bca280daf97d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ea3852c085e04bd283fc75603f24a124'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:42.409708 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-56af5a89-2055-4cb9-9eee-bca280daf97d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ea3852c085e04bd283fc75603f24a124'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:42.411075 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-e446c267-68c8-4db7-9286-19a45eb14065 None None] MFA Rules not processed for user `794ca389002d41748d7925216ed964a5`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:00:42.419429 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-e446c267-68c8-4db7-9286-19a45eb14065 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:42.428481 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 689/1359] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:00:41 2022] DELETE /identity/v3/users/ea3852c085e04bd283fc75603f24a124 => generated 0 bytes in 549 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:00:42.429662 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56af5a89-2055-4cb9-9eee-bca280daf97d admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:42.437732 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-56af5a89-2055-4cb9-9eee-bca280daf97d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:42.666568 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 671/1360] 10.222.0.36 () {58 vars in 1009 bytes} [Thu Oct 20 09:00:42 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 318 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:00:42.682889 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e446c267-68c8-4db7-9286-19a45eb14065 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:42.685693 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-e446c267-68c8-4db7-9286-19a45eb14065 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:42.701902 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56af5a89-2055-4cb9-9eee-bca280daf97d admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:42.709133 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-053b8f97-3e33-4b46-aa53-7af382e0d445 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:42.710920 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-053b8f97-3e33-4b46-aa53-7af382e0d445 service neutron] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:42.711205 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-053b8f97-3e33-4b46-aa53-7af382e0d445 service neutron] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:42.711483 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-053b8f97-3e33-4b46-aa53-7af382e0d445 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:42.716992 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e446c267-68c8-4db7-9286-19a45eb14065 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:42.720441 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-053b8f97-3e33-4b46-aa53-7af382e0d445 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:42.726164 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a72f47fc-80a8-4102-9f0a-ae23d31a901c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:42.727838 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72f47fc-80a8-4102-9f0a-ae23d31a901c service neutron] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:42.728234 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72f47fc-80a8-4102-9f0a-ae23d31a901c service neutron] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:42.728640 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72f47fc-80a8-4102-9f0a-ae23d31a901c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:42.735866 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-053b8f97-3e33-4b46-aa53-7af382e0d445 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:42.737443 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-053b8f97-3e33-4b46-aa53-7af382e0d445 service neutron] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:42.739482 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a72f47fc-80a8-4102-9f0a-ae23d31a901c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:42.743672 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a72f47fc-80a8-4102-9f0a-ae23d31a901c service neutron] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:42.757453 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 672/1361] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:00:42 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 75 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:00:42.758567 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 690/1362] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:00:42 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 329 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:00:43.372346 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a72f47fc-80a8-4102-9f0a-ae23d31a901c service neutron] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:43.376123 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-a72f47fc-80a8-4102-9f0a-ae23d31a901c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:43.620737 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a72f47fc-80a8-4102-9f0a-ae23d31a901c service neutron] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:43.629827 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cba21e13-32e4-4cb2-9a16-00f301e91530 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:43.630774 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba21e13-32e4-4cb2-9a16-00f301e91530 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:43.631207 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba21e13-32e4-4cb2-9a16-00f301e91530 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:43.631638 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba21e13-32e4-4cb2-9a16-00f301e91530 admin admin] PATH_INFO: `/v3/projects/838d5093ccd149ce882473932672a6bf` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:43.632964 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cba21e13-32e4-4cb2-9a16-00f301e91530 admin admin] RBAC: Authorizing `identity:delete_project(project_id=838d5093ccd149ce882473932672a6bf)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:43.636578 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cba21e13-32e4-4cb2-9a16-00f301e91530 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:43.755944 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:00:43.779708 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-cba21e13-32e4-4cb2-9a16-00f301e91530 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '838d5093ccd149ce882473932672a6bf', 'request_id': 'req-cba21e13-32e4-4cb2-9a16-00f301e91530'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:43.799504 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-cba21e13-32e4-4cb2-9a16-00f301e91530 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '838d5093ccd149ce882473932672a6bf', 'request_id': 'req-cba21e13-32e4-4cb2-9a16-00f301e91530'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:43.826533 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-cba21e13-32e4-4cb2-9a16-00f301e91530 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '838d5093ccd149ce882473932672a6bf', 'request_id': 'req-cba21e13-32e4-4cb2-9a16-00f301e91530'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:43.859468 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-cba21e13-32e4-4cb2-9a16-00f301e91530 admin admin] The token cache is being invalidate because project 838d5093ccd149ce882473932672a6bf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81561) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:00:43.859770 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-cba21e13-32e4-4cb2-9a16-00f301e91530 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:43.865082 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-cba21e13-32e4-4cb2-9a16-00f301e91530 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:43.867755 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 673/1363] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:00:43 2022] DELETE /identity/v3/projects/838d5093ccd149ce882473932672a6bf => generated 0 bytes in 496 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:00:43.874722 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26fdaf09-a486-45db-9516-b2db7f5df770 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:43.875715 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26fdaf09-a486-45db-9516-b2db7f5df770 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:43.876147 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26fdaf09-a486-45db-9516-b2db7f5df770 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:43.916697 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-26fdaf09-a486-45db-9516-b2db7f5df770 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:00:43.923206 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cba21e13-32e4-4cb2-9a16-00f301e91530 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:43.925816 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-cba21e13-32e4-4cb2-9a16-00f301e91530 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:43.926039 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-26fdaf09-a486-45db-9516-b2db7f5df770 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:43.980537 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:00:44.114967 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 691/1364] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:00:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 241 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:00:44.116204 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26fdaf09-a486-45db-9516-b2db7f5df770 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:44.123425 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-26fdaf09-a486-45db-9516-b2db7f5df770 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:44.169338 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cba21e13-32e4-4cb2-9a16-00f301e91530 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:44.175303 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-828881e3-1e1a-47cf-b549-249ffb889c6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:44.176546 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-828881e3-1e1a-47cf-b549-249ffb889c6a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:44.177093 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-828881e3-1e1a-47cf-b549-249ffb889c6a admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:44.177605 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-828881e3-1e1a-47cf-b549-249ffb889c6a admin admin] PATH_INFO: `/v3/users/8fbb2f982ece43a289217e5a6c16f3ab` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:44.179981 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-828881e3-1e1a-47cf-b549-249ffb889c6a admin admin] RBAC: Authorizing `identity:delete_user(user_id=8fbb2f982ece43a289217e5a6c16f3ab)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:44.192886 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-828881e3-1e1a-47cf-b549-249ffb889c6a admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:44.326078 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26fdaf09-a486-45db-9516-b2db7f5df770 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:44.366382 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d43ac9de-12ba-4955-8c30-2dea0b141649 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a94e868115ec49918eda10724263930c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:44.368574 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d43ac9de-12ba-4955-8c30-2dea0b141649 service nova] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:44.374346 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d43ac9de-12ba-4955-8c30-2dea0b141649 service nova] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:44.374700 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d43ac9de-12ba-4955-8c30-2dea0b141649 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:44.387833 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-d43ac9de-12ba-4955-8c30-2dea0b141649 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:44.403101 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-828881e3-1e1a-47cf-b549-249ffb889c6a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8fbb2f982ece43a289217e5a6c16f3ab'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:44.417559 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-828881e3-1e1a-47cf-b549-249ffb889c6a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8fbb2f982ece43a289217e5a6c16f3ab'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:44.475656 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-828881e3-1e1a-47cf-b549-249ffb889c6a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8fbb2f982ece43a289217e5a6c16f3ab'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:44.493756 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-828881e3-1e1a-47cf-b549-249ffb889c6a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8fbb2f982ece43a289217e5a6c16f3ab'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:44.494683 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-828881e3-1e1a-47cf-b549-249ffb889c6a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8fbb2f982ece43a289217e5a6c16f3ab'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:44.506184 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 674/1365] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:00:43 2022] DELETE /identity/v3/users/8fbb2f982ece43a289217e5a6c16f3ab => generated 0 bytes in 582 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:00:44.507935 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7d933a9-37e5-46dd-83c3-afd1a28a2b93 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:44.508498 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7d933a9-37e5-46dd-83c3-afd1a28a2b93 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:44.520910 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7d933a9-37e5-46dd-83c3-afd1a28a2b93 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:44.532603 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d43ac9de-12ba-4955-8c30-2dea0b141649 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:44.535035 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d43ac9de-12ba-4955-8c30-2dea0b141649 service nova] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:44.556225 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 692/1366] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:00:44 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 440 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:00:44.560364 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d43ac9de-12ba-4955-8c30-2dea0b141649 service nova] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:44.567093 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-d43ac9de-12ba-4955-8c30-2dea0b141649 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:44.595471 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-e7d933a9-37e5-46dd-83c3-afd1a28a2b93 None None] MFA Rules not processed for user `8002ab26f6d64822aaa65aa730781b76`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:00:44.596697 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-e7d933a9-37e5-46dd-83c3-afd1a28a2b93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:44.697091 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:00:44.751503 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:00:44.764585 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d43ac9de-12ba-4955-8c30-2dea0b141649 service nova] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:44.819626 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 675/1367] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:00:44 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 313 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:00:44.829450 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b3cd2c-48e3-4166-9998-002f9a3dbf2a None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:44.829752 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b3cd2c-48e3-4166-9998-002f9a3dbf2a None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:44.830035 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b3cd2c-48e3-4166-9998-002f9a3dbf2a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:44.891551 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-50b3cd2c-48e3-4166-9998-002f9a3dbf2a None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:00:44.899037 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-50b3cd2c-48e3-4166-9998-002f9a3dbf2a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:45.030173 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34eb35b8-448d-4527-9603-3e7767c43f1e service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ad03ab76b9c44b1cb369475afa25eddd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:45.030830 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34eb35b8-448d-4527-9603-3e7767c43f1e service placement] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:45.031110 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34eb35b8-448d-4527-9603-3e7767c43f1e service placement] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:45.031379 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34eb35b8-448d-4527-9603-3e7767c43f1e service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:45.036838 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-34eb35b8-448d-4527-9603-3e7767c43f1e service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:45.079794 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34eb35b8-448d-4527-9603-3e7767c43f1e service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:45.082031 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34eb35b8-448d-4527-9603-3e7767c43f1e service placement] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:45.094105 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 676/1368] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:00:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 274 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:00:45.100710 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f05325-5e22-481a-aba1-582a4a0617b5 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:45.101112 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f05325-5e22-481a-aba1-582a4a0617b5 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:45.101395 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f05325-5e22-481a-aba1-582a4a0617b5 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:45.104703 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 677/1369] 10.222.0.36 () {54 vars in 961 bytes} [Thu Oct 20 09:00:45 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 09:00:45.105958 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45255d85-a5d5-46f5-be94-917e1c3455ca None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:45.106320 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45255d85-a5d5-46f5-be94-917e1c3455ca None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:45.106627 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45255d85-a5d5-46f5-be94-917e1c3455ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:45.121537 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 693/1370] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:00:44 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 565 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:00:45.123194 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd3c1b2-66be-4da0-a216-af313857f0f7 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:45.123618 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd3c1b2-66be-4da0-a216-af313857f0f7 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:45.124027 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd3c1b2-66be-4da0-a216-af313857f0f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:45.150017 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-45255d85-a5d5-46f5-be94-917e1c3455ca None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:00:45.151915 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-45255d85-a5d5-46f5-be94-917e1c3455ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:45.161784 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 678/1371] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:00:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 56 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:00:45.162987 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45255d85-a5d5-46f5-be94-917e1c3455ca None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:45.173430 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-5cd3c1b2-66be-4da0-a216-af313857f0f7 None None] MFA Rules not processed for user `7f932695a5994c5790d14c210ea65326`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:00:45.174902 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-45255d85-a5d5-46f5-be94-917e1c3455ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:45.175599 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-5cd3c1b2-66be-4da0-a216-af313857f0f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:45.323471 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 694/1372] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:00:45 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:00:45.323471 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cd3c1b2-66be-4da0-a216-af313857f0f7 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:45.380800 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cd3c1b2-66be-4da0-a216-af313857f0f7 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:45.411473 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8168fce6-9451-4919-9986-1428727dbe57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:45.416780 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8168fce6-9451-4919-9986-1428727dbe57 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:45.419631 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8168fce6-9451-4919-9986-1428727dbe57 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:45.419631 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8168fce6-9451-4919-9986-1428727dbe57 admin admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:45.420465 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8168fce6-9451-4919-9986-1428727dbe57 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 09:00:45.420601 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8168fce6-9451-4919-9986-1428727dbe57 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:45.432874 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8168fce6-9451-4919-9986-1428727dbe57 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:45.505026 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 695/1373] 10.222.0.36 () {58 vars in 1153 bytes} [Thu Oct 20 09:00:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 09:00:45.511558 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5836b9-f1f6-46d1-816d-f53a03454060 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:45.511897 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5836b9-f1f6-46d1-816d-f53a03454060 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:45.512216 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5836b9-f1f6-46d1-816d-f53a03454060 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:45.566985 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-1f5836b9-f1f6-46d1-816d-f53a03454060 None None] MFA Rules not processed for user `ad03ab76b9c44b1cb369475afa25eddd`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:00:45.568257 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-1f5836b9-f1f6-46d1-816d-f53a03454060 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:45.598042 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 696/1374] 10.222.0.36 () {58 vars in 1058 bytes} [Thu Oct 20 09:00:45 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:00:45.599856 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f5836b9-f1f6-46d1-816d-f53a03454060 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:45.601693 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-1f5836b9-f1f6-46d1-816d-f53a03454060 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:45.660995 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45255d85-a5d5-46f5-be94-917e1c3455ca None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:45.676683 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ada99a8e-0bd8-4ffc-b248-9919e5c65892 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a94e868115ec49918eda10724263930c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:45.676683 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada99a8e-0bd8-4ffc-b248-9919e5c65892 service nova] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:45.676683 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada99a8e-0bd8-4ffc-b248-9919e5c65892 service nova] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:45.676683 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada99a8e-0bd8-4ffc-b248-9919e5c65892 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:45.702500 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ada99a8e-0bd8-4ffc-b248-9919e5c65892 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:45.705956 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ada99a8e-0bd8-4ffc-b248-9919e5c65892 service nova] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:45.759263 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 679/1375] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:00:45 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 597 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:00:45.762806 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f5836b9-f1f6-46d1-816d-f53a03454060 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:45.763400 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ada99a8e-0bd8-4ffc-b248-9919e5c65892 service nova] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:45.778063 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b68503-39ed-406f-84d5-15683f0d3cf1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:45.778739 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b68503-39ed-406f-84d5-15683f0d3cf1 service neutron] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:45.779019 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b68503-39ed-406f-84d5-15683f0d3cf1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:45.779294 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b68503-39ed-406f-84d5-15683f0d3cf1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:45.801421 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3b68503-39ed-406f-84d5-15683f0d3cf1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:45.804760 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3b68503-39ed-406f-84d5-15683f0d3cf1 service neutron] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:45.818228 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ada99a8e-0bd8-4ffc-b248-9919e5c65892 service nova] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:45.835103 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e748e790-4426-4ba0-950f-46b9c6d0aad7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:45.835955 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e748e790-4426-4ba0-950f-46b9c6d0aad7 admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:45.836397 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e748e790-4426-4ba0-950f-46b9c6d0aad7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:45.836864 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e748e790-4426-4ba0-950f-46b9c6d0aad7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:45.839159 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e748e790-4426-4ba0-950f-46b9c6d0aad7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:45.842919 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 697/1376] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:00:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 244 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:00:45.843906 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b68503-39ed-406f-84d5-15683f0d3cf1 service neutron] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:45.844725 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e748e790-4426-4ba0-950f-46b9c6d0aad7 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:45.881521 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b68503-39ed-406f-84d5-15683f0d3cf1 service neutron] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:45.903177 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3bc77ae-afc5-4935-920c-6f219b142672 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ad03ab76b9c44b1cb369475afa25eddd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:45.906826 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3bc77ae-afc5-4935-920c-6f219b142672 service placement] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:45.907115 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3bc77ae-afc5-4935-920c-6f219b142672 service placement] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:45.907812 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3bc77ae-afc5-4935-920c-6f219b142672 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:45.944685 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3bc77ae-afc5-4935-920c-6f219b142672 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:45.947099 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3bc77ae-afc5-4935-920c-6f219b142672 service placement] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:45.966457 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 680/1377] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:00:45 2022] POST /identity/v3/projects => generated 385 bytes in 205 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:00:45.968614 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e748e790-4426-4ba0-950f-46b9c6d0aad7 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:46.003432 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 698/1378] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:00:45 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 160 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:00:46.056946 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:00:46.125044 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e748e790-4426-4ba0-950f-46b9c6d0aad7 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:46.139134 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a8a970-519f-459f-b2ec-61677baeff8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:46.139804 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a8a970-519f-459f-b2ec-61677baeff8e admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:46.140030 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a8a970-519f-459f-b2ec-61677baeff8e admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:46.140298 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a8a970-519f-459f-b2ec-61677baeff8e admin admin] PATH_INFO: `/v3/users` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:46.140939 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85a8a970-519f-459f-b2ec-61677baeff8e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:46.142993 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85a8a970-519f-459f-b2ec-61677baeff8e admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:46.185075 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 681/1379] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:00:45 2022] POST /identity/v3/users => generated 293 bytes in 218 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:00:46.188838 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3bc77ae-afc5-4935-920c-6f219b142672 service placement] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:46.203524 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3bc77ae-afc5-4935-920c-6f219b142672 service placement] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:46.220234 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d3fc5d-28cd-4a2c-a75a-76bba5596ade admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:46.220857 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d3fc5d-28cd-4a2c-a75a-76bba5596ade admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:46.221137 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d3fc5d-28cd-4a2c-a75a-76bba5596ade admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:46.221409 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d3fc5d-28cd-4a2c-a75a-76bba5596ade admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:46.222054 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05d3fc5d-28cd-4a2c-a75a-76bba5596ade admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:46.224046 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05d3fc5d-28cd-4a2c-a75a-76bba5596ade admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:46.225447 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:00:46.250318 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 699/1380] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:00:46 2022] GET /identity/v3/roles => generated 1420 bytes in 60 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:00:46.252623 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a8a970-519f-459f-b2ec-61677baeff8e admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:46.273948 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a8a970-519f-459f-b2ec-61677baeff8e admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:46.287156 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d793c7b-464e-449c-9470-0bcbba255caf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:46.287771 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d793c7b-464e-449c-9470-0bcbba255caf admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:46.288053 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d793c7b-464e-449c-9470-0bcbba255caf admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:46.288319 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d793c7b-464e-449c-9470-0bcbba255caf admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:46.288980 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d793c7b-464e-449c-9470-0bcbba255caf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:46.303312 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d793c7b-464e-449c-9470-0bcbba255caf admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:46.321868 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 682/1381] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:00:46 2022] GET /identity/v3/roles => generated 1420 bytes in 70 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:00:46.326526 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d3fc5d-28cd-4a2c-a75a-76bba5596ade admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:46.347561 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d3fc5d-28cd-4a2c-a75a-76bba5596ade admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:46.354818 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deda06fd-4dfa-4844-8fc6-efa4581d1d2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:46.356817 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deda06fd-4dfa-4844-8fc6-efa4581d1d2d admin admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:46.357069 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deda06fd-4dfa-4844-8fc6-efa4581d1d2d admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:46.358364 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deda06fd-4dfa-4844-8fc6-efa4581d1d2d admin admin] PATH_INFO: `/v3/projects/fead5997db8244ac9b7cc96e21eb7178/users/e403fe6629684850bc6cda6664ba3852/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:46.389792 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deda06fd-4dfa-4844-8fc6-efa4581d1d2d admin admin] RBAC: Authorizing `identity:create_grant(project_id=fead5997db8244ac9b7cc96e21eb7178, user_id=e403fe6629684850bc6cda6664ba3852, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:46.394215 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deda06fd-4dfa-4844-8fc6-efa4581d1d2d admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:46.439619 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 700/1382] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:00:46 2022] PUT /identity/v3/projects/fead5997db8244ac9b7cc96e21eb7178/users/e403fe6629684850bc6cda6664ba3852/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:00:46.440793 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9014db60-858c-44a1-b4f1-9ac73befc3b8 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:46.441086 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9014db60-858c-44a1-b4f1-9ac73befc3b8 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:46.441365 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9014db60-858c-44a1-b4f1-9ac73befc3b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:46.511585 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-9014db60-858c-44a1-b4f1-9ac73befc3b8 None None] MFA Rules not processed for user `e403fe6629684850bc6cda6664ba3852`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:00:46.512850 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-9014db60-858c-44a1-b4f1-9ac73befc3b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:46.662874 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 683/1383] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:00:46 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 223 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:00:46.668102 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deda06fd-4dfa-4844-8fc6-efa4581d1d2d admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:46.849081 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deda06fd-4dfa-4844-8fc6-efa4581d1d2d admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:46.865628 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de5f2b47-7c32-462b-aebe-3953a1449704 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:46.869592 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5f2b47-7c32-462b-aebe-3953a1449704 admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:46.870827 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5f2b47-7c32-462b-aebe-3953a1449704 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:46.871087 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5f2b47-7c32-462b-aebe-3953a1449704 admin admin] PATH_INFO: `/v3/projects` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:46.874280 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de5f2b47-7c32-462b-aebe-3953a1449704 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:46.879977 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de5f2b47-7c32-462b-aebe-3953a1449704 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:46.969005 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 701/1384] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:00:46 2022] POST /identity/v3/projects => generated 387 bytes in 302 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:00:46.969313 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9014db60-858c-44a1-b4f1-9ac73befc3b8 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:46.985483 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9014db60-858c-44a1-b4f1-9ac73befc3b8 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:47.116451 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de5f2b47-7c32-462b-aebe-3953a1449704 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:47.120035 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-de5f2b47-7c32-462b-aebe-3953a1449704 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:47.155395 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ca0fadd-7c3e-4fe2-92ca-b529b898b140 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:47.156210 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca0fadd-7c3e-4fe2-92ca-b529b898b140 admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:47.156627 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca0fadd-7c3e-4fe2-92ca-b529b898b140 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:47.157045 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca0fadd-7c3e-4fe2-92ca-b529b898b140 admin admin] PATH_INFO: `/v3/users` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:47.157917 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ca0fadd-7c3e-4fe2-92ca-b529b898b140 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:47.160070 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ca0fadd-7c3e-4fe2-92ca-b529b898b140 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:47.240895 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 684/1385] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:00:46 2022] POST /identity/v3/users => generated 300 bytes in 272 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:00:47.241969 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ca0fadd-7c3e-4fe2-92ca-b529b898b140 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:47.265059 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de5f2b47-7c32-462b-aebe-3953a1449704 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:47.272247 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09ac18ce-0e3a-41e1-b75d-d49ac8276748 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:47.272499 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ca0fadd-7c3e-4fe2-92ca-b529b898b140 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:47.273257 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ac18ce-0e3a-41e1-b75d-d49ac8276748 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:47.273800 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ac18ce-0e3a-41e1-b75d-d49ac8276748 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:47.274335 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ac18ce-0e3a-41e1-b75d-d49ac8276748 admin admin] PATH_INFO: `/v3/projects/16dda493652d4c87b075ee8cef7a8a95` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:47.275732 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09ac18ce-0e3a-41e1-b75d-d49ac8276748 admin admin] RBAC: Authorizing `identity:delete_project(project_id=16dda493652d4c87b075ee8cef7a8a95)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:47.278555 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09ac18ce-0e3a-41e1-b75d-d49ac8276748 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:47.293099 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f6b46db-3e2f-4edd-934d-f3cffc956c7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:47.294330 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f6b46db-3e2f-4edd-934d-f3cffc956c7f admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:47.294330 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f6b46db-3e2f-4edd-934d-f3cffc956c7f admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:47.294330 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f6b46db-3e2f-4edd-934d-f3cffc956c7f admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:47.298999 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f6b46db-3e2f-4edd-934d-f3cffc956c7f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:47.300666 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f6b46db-3e2f-4edd-934d-f3cffc956c7f admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:47.334071 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 685/1386] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:00:47 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:00:47.335010 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f6b46db-3e2f-4edd-934d-f3cffc956c7f admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:47.386861 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f6b46db-3e2f-4edd-934d-f3cffc956c7f admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:47.415806 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad95398b-6450-411b-b540-8a05742c3353 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:47.416220 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-09ac18ce-0e3a-41e1-b75d-d49ac8276748 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '16dda493652d4c87b075ee8cef7a8a95', 'request_id': 'req-09ac18ce-0e3a-41e1-b75d-d49ac8276748'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:47.426686 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad95398b-6450-411b-b540-8a05742c3353 admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:47.426915 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-09ac18ce-0e3a-41e1-b75d-d49ac8276748 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '16dda493652d4c87b075ee8cef7a8a95', 'request_id': 'req-09ac18ce-0e3a-41e1-b75d-d49ac8276748'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:47.431423 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad95398b-6450-411b-b540-8a05742c3353 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:47.431987 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad95398b-6450-411b-b540-8a05742c3353 admin admin] PATH_INFO: `/v3/projects/c2b7484f3ae34596bae4794f348cd875/users/ab7b646274c34e4089ebe5ccd7511ed3/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:47.463685 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=93,keystone:INSERT=2,keystone:DELETE=5 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:00:47.469900 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-09ac18ce-0e3a-41e1-b75d-d49ac8276748 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '16dda493652d4c87b075ee8cef7a8a95', 'request_id': 'req-09ac18ce-0e3a-41e1-b75d-d49ac8276748'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:47.488570 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad95398b-6450-411b-b540-8a05742c3353 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c2b7484f3ae34596bae4794f348cd875, user_id=ab7b646274c34e4089ebe5ccd7511ed3, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:47.505373 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-09ac18ce-0e3a-41e1-b75d-d49ac8276748 admin admin] The token cache is being invalidate because project 16dda493652d4c87b075ee8cef7a8a95 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81562) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:00:47.505904 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad95398b-6450-411b-b540-8a05742c3353 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:47.506272 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-09ac18ce-0e3a-41e1-b75d-d49ac8276748 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:47.509048 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-09ac18ce-0e3a-41e1-b75d-d49ac8276748 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:47.514412 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 702/1387] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:00:47 2022] DELETE /identity/v3/projects/16dda493652d4c87b075ee8cef7a8a95 => generated 0 bytes in 398 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:00:47.519464 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c47d1c-92f3-43f5-836b-d47dc3a2cb56 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:47.519754 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c47d1c-92f3-43f5-836b-d47dc3a2cb56 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:47.520167 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c47d1c-92f3-43f5-836b-d47dc3a2cb56 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:47.546278 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:00:47.590043 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 686/1388] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:00:47 2022] PUT /identity/v3/projects/c2b7484f3ae34596bae4794f348cd875/users/ab7b646274c34e4089ebe5ccd7511ed3/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 252 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:00:47.591848 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad95398b-6450-411b-b540-8a05742c3353 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:47.595937 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-ad95398b-6450-411b-b540-8a05742c3353 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:47.599623 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-50c47d1c-92f3-43f5-836b-d47dc3a2cb56 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:00:47.606775 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-50c47d1c-92f3-43f5-836b-d47dc3a2cb56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:47.839126 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad95398b-6450-411b-b540-8a05742c3353 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:47.854207 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-123e88b5-d5d2-4dea-a5a2-e624b1e7e70d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:47.855314 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-123e88b5-d5d2-4dea-a5a2-e624b1e7e70d admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:47.855765 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-123e88b5-d5d2-4dea-a5a2-e624b1e7e70d admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:47.856184 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-123e88b5-d5d2-4dea-a5a2-e624b1e7e70d admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:47.857042 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-123e88b5-d5d2-4dea-a5a2-e624b1e7e70d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:47.860409 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-123e88b5-d5d2-4dea-a5a2-e624b1e7e70d admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:47.878050 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 703/1389] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:00:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 360 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:00:47.889689 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 687/1390] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:00:47 2022] GET /identity/v3/roles => generated 1420 bytes in 296 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 20 09:00:47.890206 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064f7b05-2f9c-40ac-8e50-acd07566f02b None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:47.890521 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064f7b05-2f9c-40ac-8e50-acd07566f02b None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:47.890791 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064f7b05-2f9c-40ac-8e50-acd07566f02b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:47.891232 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-123e88b5-d5d2-4dea-a5a2-e624b1e7e70d admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:47.909210 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-123e88b5-d5d2-4dea-a5a2-e624b1e7e70d admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:47.920535 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04812a32-37cb-47e7-8f00-0f5bafee7088 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:47.922572 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04812a32-37cb-47e7-8f00-0f5bafee7088 admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:47.922854 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04812a32-37cb-47e7-8f00-0f5bafee7088 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:47.923206 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04812a32-37cb-47e7-8f00-0f5bafee7088 admin admin] PATH_INFO: `/v3/projects/c2b7484f3ae34596bae4794f348cd875/users/ab7b646274c34e4089ebe5ccd7511ed3/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:47.931920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04812a32-37cb-47e7-8f00-0f5bafee7088 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c2b7484f3ae34596bae4794f348cd875, user_id=ab7b646274c34e4089ebe5ccd7511ed3, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:47.936627 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04812a32-37cb-47e7-8f00-0f5bafee7088 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:47.953711 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-064f7b05-2f9c-40ac-8e50-acd07566f02b None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:00:47.956581 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-064f7b05-2f9c-40ac-8e50-acd07566f02b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:47.968929 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 688/1391] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:00:47 2022] PUT /identity/v3/projects/c2b7484f3ae34596bae4794f348cd875/users/ab7b646274c34e4089ebe5ccd7511ed3/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 78 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:00:47.983660 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06a0173-4a85-47e2-ad0d-16c6ad1d710b None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:47.983972 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06a0173-4a85-47e2-ad0d-16c6ad1d710b None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:47.984239 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06a0173-4a85-47e2-ad0d-16c6ad1d710b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:48.171714 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-d06a0173-4a85-47e2-ad0d-16c6ad1d710b None None] MFA Rules not processed for user `ab7b646274c34e4089ebe5ccd7511ed3`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:00:48.176174 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-d06a0173-4a85-47e2-ad0d-16c6ad1d710b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:48.277100 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 704/1392] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:00:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 386 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:00:48.280179 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-064f7b05-2f9c-40ac-8e50-acd07566f02b None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:48.309082 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-064f7b05-2f9c-40ac-8e50-acd07566f02b None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:48.329173 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6698ae9-acf4-4413-8483-e3cab3362336 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:48.329803 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6698ae9-acf4-4413-8483-e3cab3362336 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:48.330080 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6698ae9-acf4-4413-8483-e3cab3362336 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:48.330350 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6698ae9-acf4-4413-8483-e3cab3362336 admin admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:48.333706 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6698ae9-acf4-4413-8483-e3cab3362336 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 09:00:48.334014 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6698ae9-acf4-4413-8483-e3cab3362336 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:48.339622 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6698ae9-acf4-4413-8483-e3cab3362336 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:48.376903 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 705/1393] 10.222.0.36 () {58 vars in 1153 bytes} [Thu Oct 20 09:00:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 97 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 09:00:48.377674 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6698ae9-acf4-4413-8483-e3cab3362336 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:48.452717 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6698ae9-acf4-4413-8483-e3cab3362336 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:48.462208 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 689/1394] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:00:47 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 475 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:00:48.470342 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63f7d43d-3ab9-4810-a44d-a5c4d9e93f04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:48.471068 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f7d43d-3ab9-4810-a44d-a5c4d9e93f04 admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:48.471419 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f7d43d-3ab9-4810-a44d-a5c4d9e93f04 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:48.471758 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f7d43d-3ab9-4810-a44d-a5c4d9e93f04 admin admin] PATH_INFO: `/v3/projects` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:48.472636 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63f7d43d-3ab9-4810-a44d-a5c4d9e93f04 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:48.477030 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63f7d43d-3ab9-4810-a44d-a5c4d9e93f04 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:48.536451 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 706/1395] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:00:48 2022] POST /identity/v3/projects => generated 363 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:00:48.542967 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d06a0173-4a85-47e2-ad0d-16c6ad1d710b None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:48.563829 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d06a0173-4a85-47e2-ad0d-16c6ad1d710b None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:48.626036 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63f7d43d-3ab9-4810-a44d-a5c4d9e93f04 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:48.628654 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-63f7d43d-3ab9-4810-a44d-a5c4d9e93f04 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:48.782847 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63f7d43d-3ab9-4810-a44d-a5c4d9e93f04 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:48.797454 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d0a1f3e-ec1d-4f52-adeb-da7aae8cb7cb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:48.798108 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d0a1f3e-ec1d-4f52-adeb-da7aae8cb7cb service cinder] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:48.798390 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d0a1f3e-ec1d-4f52-adeb-da7aae8cb7cb service cinder] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:48.798692 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d0a1f3e-ec1d-4f52-adeb-da7aae8cb7cb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:48.814866 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcb0783e-b0b5-4869-ab9e-fbf879c266d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:48.816933 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb0783e-b0b5-4869-ab9e-fbf879c266d6 admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:48.817312 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb0783e-b0b5-4869-ab9e-fbf879c266d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:48.817653 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb0783e-b0b5-4869-ab9e-fbf879c266d6 admin admin] PATH_INFO: `/v3/users` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:48.818133 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcb0783e-b0b5-4869-ab9e-fbf879c266d6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:48.818303 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:DELETE=2,keystone:SELECT=96,keystone:INSERT=2 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:00:48.830948 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcb0783e-b0b5-4869-ab9e-fbf879c266d6 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:48.864596 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:00:48.884677 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 690/1396] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:00:48 2022] POST /identity/v3/users => generated 282 bytes in 345 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:00:48.889475 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcb0783e-b0b5-4869-ab9e-fbf879c266d6 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:48.918919 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcb0783e-b0b5-4869-ab9e-fbf879c266d6 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:48.936412 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eead895-d410-43fd-8eff-f70992de7832 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:48.939957 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eead895-d410-43fd-8eff-f70992de7832 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:48.940262 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eead895-d410-43fd-8eff-f70992de7832 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:48.943428 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eead895-d410-43fd-8eff-f70992de7832 admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:48.945445 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eead895-d410-43fd-8eff-f70992de7832 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:48.955518 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eead895-d410-43fd-8eff-f70992de7832 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:49.008950 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 691/1397] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:00:48 2022] GET /identity/v3/roles => generated 1420 bytes in 118 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 20 09:00:49.011076 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eead895-d410-43fd-8eff-f70992de7832 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:49.037709 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eead895-d410-43fd-8eff-f70992de7832 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:49.053646 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aab76298-7405-4673-810f-d6413a9f867f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:49.054286 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab76298-7405-4673-810f-d6413a9f867f admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:49.054729 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab76298-7405-4673-810f-d6413a9f867f admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:49.055011 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab76298-7405-4673-810f-d6413a9f867f admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:49.057228 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aab76298-7405-4673-810f-d6413a9f867f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:49.062698 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aab76298-7405-4673-810f-d6413a9f867f admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:49.068276 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d0a1f3e-ec1d-4f52-adeb-da7aae8cb7cb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:49.070103 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d0a1f3e-ec1d-4f52-adeb-da7aae8cb7cb service cinder] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:49.093661 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 707/1398] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:00:48 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 468 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 20 09:00:49.121221 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d0a1f3e-ec1d-4f52-adeb-da7aae8cb7cb service cinder] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:49.123670 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 692/1399] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:00:49 2022] GET /identity/v3/roles => generated 1420 bytes in 113 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:00:49.137487 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d0a1f3e-ec1d-4f52-adeb-da7aae8cb7cb service cinder] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:49.146982 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ded5deb-dff4-407a-a2a0-5b13ee1f2499 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:49.147296 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ded5deb-dff4-407a-a2a0-5b13ee1f2499 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:49.147541 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ded5deb-dff4-407a-a2a0-5b13ee1f2499 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:49.158453 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b0bc3fe-20bd-4dbd-94cc-4eebcbc522aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:49.161651 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b0bc3fe-20bd-4dbd-94cc-4eebcbc522aa admin admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:49.167883 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b0bc3fe-20bd-4dbd-94cc-4eebcbc522aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:49.170965 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b0bc3fe-20bd-4dbd-94cc-4eebcbc522aa admin admin] PATH_INFO: `/v3/projects/a0542d0ca1eb4be69dd0b8b3ab26790a/users/a3bb6bab4d7646cca07529be996c86b8/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:49.223417 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b0bc3fe-20bd-4dbd-94cc-4eebcbc522aa admin admin] RBAC: Authorizing `identity:create_grant(project_id=a0542d0ca1eb4be69dd0b8b3ab26790a, user_id=a3bb6bab4d7646cca07529be996c86b8, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:49.230252 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-9ded5deb-dff4-407a-a2a0-5b13ee1f2499 None None] MFA Rules not processed for user `fd9733d1bb2b4520b70090542615ddf9`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:00:49.235573 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-9ded5deb-dff4-407a-a2a0-5b13ee1f2499 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:49.235928 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b0bc3fe-20bd-4dbd-94cc-4eebcbc522aa admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:49.288810 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 708/1400] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:00:49 2022] PUT /identity/v3/projects/a0542d0ca1eb4be69dd0b8b3ab26790a/users/a3bb6bab4d7646cca07529be996c86b8/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:00:49.293783 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6cefaca-f3e4-4341-8a1f-31b8052c8194 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:49.293952 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6cefaca-f3e4-4341-8a1f-31b8052c8194 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:49.294224 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6cefaca-f3e4-4341-8a1f-31b8052c8194 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:49.389484 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-b6cefaca-f3e4-4341-8a1f-31b8052c8194 None None] MFA Rules not processed for user `a3bb6bab4d7646cca07529be996c86b8`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:00:49.390771 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-b6cefaca-f3e4-4341-8a1f-31b8052c8194 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:49.553585 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 693/1401] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:00:49 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 408 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 20 09:00:49.558015 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ded5deb-dff4-407a-a2a0-5b13ee1f2499 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:49.617301 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ded5deb-dff4-407a-a2a0-5b13ee1f2499 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:49.674106 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 709/1402] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:00:49 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 384 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:00:49.676993 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6cefaca-f3e4-4341-8a1f-31b8052c8194 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:49.833008 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb55bcc-442a-48a2-a9d3-ecda7eba8d80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:49.833734 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb55bcc-442a-48a2-a9d3-ecda7eba8d80 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:49.834025 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb55bcc-442a-48a2-a9d3-ecda7eba8d80 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:49.834300 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb55bcc-442a-48a2-a9d3-ecda7eba8d80 admin admin] PATH_INFO: `/v3/users/e403fe6629684850bc6cda6664ba3852` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:49.842952 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eb55bcc-442a-48a2-a9d3-ecda7eba8d80 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e403fe6629684850bc6cda6664ba3852)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:49.844937 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eb55bcc-442a-48a2-a9d3-ecda7eba8d80 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:49.932137 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6cefaca-f3e4-4341-8a1f-31b8052c8194 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:49.944763 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a953a51-f89b-4df5-b8fe-f6b63243e2e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:49.946931 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a953a51-f89b-4df5-b8fe-f6b63243e2e1 admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:49.947633 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a953a51-f89b-4df5-b8fe-f6b63243e2e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:49.947750 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a953a51-f89b-4df5-b8fe-f6b63243e2e1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:49.949862 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a953a51-f89b-4df5-b8fe-f6b63243e2e1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:49.955660 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a953a51-f89b-4df5-b8fe-f6b63243e2e1 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:49.978389 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-7eb55bcc-442a-48a2-a9d3-ecda7eba8d80 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e403fe6629684850bc6cda6664ba3852'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:49.997765 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:00:50.001967 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-7eb55bcc-442a-48a2-a9d3-ecda7eba8d80 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e403fe6629684850bc6cda6664ba3852'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:50.038727 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-7eb55bcc-442a-48a2-a9d3-ecda7eba8d80 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e403fe6629684850bc6cda6664ba3852'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:50.043607 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 710/1403] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:00:49 2022] POST /identity/v3/projects => generated 365 bytes in 369 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:00:50.046921 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a953a51-f89b-4df5-b8fe-f6b63243e2e1 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:50.061996 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:INSERT=2,keystone:SELECT=88,keystone:DELETE=10 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:00:50.063488 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-7eb55bcc-442a-48a2-a9d3-ecda7eba8d80 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e403fe6629684850bc6cda6664ba3852'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:50.064569 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-7eb55bcc-442a-48a2-a9d3-ecda7eba8d80 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e403fe6629684850bc6cda6664ba3852'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:50.081439 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 694/1404] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:00:49 2022] DELETE /identity/v3/users/e403fe6629684850bc6cda6664ba3852 => generated 0 bytes in 527 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:00:50.089403 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d4936b-90f4-427b-a5d3-0812e4d7e35a None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:50.090222 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d4936b-90f4-427b-a5d3-0812e4d7e35a None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:50.090690 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d4936b-90f4-427b-a5d3-0812e4d7e35a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:50.131223 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-51d4936b-90f4-427b-a5d3-0812e4d7e35a None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:00:50.132673 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-51d4936b-90f4-427b-a5d3-0812e4d7e35a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:50.334416 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a953a51-f89b-4df5-b8fe-f6b63243e2e1 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:50.384207 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 695/1405] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:00:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 296 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:00:50.452929 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12aaf2d2-e8e4-4f42-a0f1-d31104669685 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:50.453247 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12aaf2d2-e8e4-4f42-a0f1-d31104669685 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:50.453505 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12aaf2d2-e8e4-4f42-a0f1-d31104669685 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:50.536754 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d02496db-a501-4443-a7f2-dd8819796104 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:50.537459 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02496db-a501-4443-a7f2-dd8819796104 admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:50.537741 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02496db-a501-4443-a7f2-dd8819796104 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:50.538026 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02496db-a501-4443-a7f2-dd8819796104 admin admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:50.538714 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d02496db-a501-4443-a7f2-dd8819796104 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:50.543780 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d02496db-a501-4443-a7f2-dd8819796104 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:50.597606 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 711/1406] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:00:50 2022] POST /identity/v3/users => generated 289 bytes in 551 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:00:50.599113 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d02496db-a501-4443-a7f2-dd8819796104 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:50.606175 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-d02496db-a501-4443-a7f2-dd8819796104 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:50.651517 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-12aaf2d2-e8e4-4f42-a0f1-d31104669685 None None] MFA Rules not processed for user `f4b0e9187ce74f7888e9e7094be5c824`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:00:50.655221 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-12aaf2d2-e8e4-4f42-a0f1-d31104669685 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:50.737621 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d02496db-a501-4443-a7f2-dd8819796104 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:50.741967 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfaa5fa3-2743-4544-90d1-53fdce9d9f58 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:50.743782 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfaa5fa3-2743-4544-90d1-53fdce9d9f58 service neutron] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:50.744189 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfaa5fa3-2743-4544-90d1-53fdce9d9f58 service neutron] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:50.744547 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfaa5fa3-2743-4544-90d1-53fdce9d9f58 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:50.755758 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfaa5fa3-2743-4544-90d1-53fdce9d9f58 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:50.757444 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfaa5fa3-2743-4544-90d1-53fdce9d9f58 service neutron] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:50.785436 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 712/1407] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:00:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 187 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:00:50.786497 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfaa5fa3-2743-4544-90d1-53fdce9d9f58 service neutron] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:50.790731 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-dfaa5fa3-2743-4544-90d1-53fdce9d9f58 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:50.815008 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfaa5fa3-2743-4544-90d1-53fdce9d9f58 service neutron] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:50.835393 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d26088d-5e6b-49ed-9476-4b6b7a666613 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:50.838164 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d26088d-5e6b-49ed-9476-4b6b7a666613 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:50.840513 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d26088d-5e6b-49ed-9476-4b6b7a666613 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:50.841391 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d26088d-5e6b-49ed-9476-4b6b7a666613 admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:50.843061 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d26088d-5e6b-49ed-9476-4b6b7a666613 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:50.852198 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d26088d-5e6b-49ed-9476-4b6b7a666613 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:50.873704 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 696/1408] 10.222.0.36 () {58 vars in 1009 bytes} [Thu Oct 20 09:00:50 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 422 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 20 09:00:50.895452 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12aaf2d2-e8e4-4f42-a0f1-d31104669685 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:50.896081 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 713/1409] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:00:50 2022] GET /identity/v3/roles => generated 1420 bytes in 110 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:00:50.910767 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12aaf2d2-e8e4-4f42-a0f1-d31104669685 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:50.928312 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e23a90f7-3a0b-43f1-a2b5-3a8acdf4ef81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:50.929265 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23a90f7-3a0b-43f1-a2b5-3a8acdf4ef81 admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:50.929719 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23a90f7-3a0b-43f1-a2b5-3a8acdf4ef81 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:50.930371 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23a90f7-3a0b-43f1-a2b5-3a8acdf4ef81 admin admin] PATH_INFO: `/v3/projects/f36ee68417014697b9c0beb7f78cbb21/users/25b4b3a05a124c5587eadcb46bc041f4/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:50.981030 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e23a90f7-3a0b-43f1-a2b5-3a8acdf4ef81 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f36ee68417014697b9c0beb7f78cbb21, user_id=25b4b3a05a124c5587eadcb46bc041f4, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:50.991252 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e23a90f7-3a0b-43f1-a2b5-3a8acdf4ef81 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:51.026872 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 697/1410] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:00:50 2022] PUT /identity/v3/projects/f36ee68417014697b9c0beb7f78cbb21/users/25b4b3a05a124c5587eadcb46bc041f4/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:00:51.034301 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d26088d-5e6b-49ed-9476-4b6b7a666613 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:51.056546 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d26088d-5e6b-49ed-9476-4b6b7a666613 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:51.138514 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:INSERT=4,keystone:SELECT=96 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:00:51.200626 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e23a90f7-3a0b-43f1-a2b5-3a8acdf4ef81 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:51.202431 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-e23a90f7-3a0b-43f1-a2b5-3a8acdf4ef81 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:51.296805 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b508ded-3cdc-4380-895e-1d2034642f3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:51.299993 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b508ded-3cdc-4380-895e-1d2034642f3a admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:51.300671 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b508ded-3cdc-4380-895e-1d2034642f3a admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:51.303933 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b508ded-3cdc-4380-895e-1d2034642f3a admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:51.305999 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b508ded-3cdc-4380-895e-1d2034642f3a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:51.309499 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b508ded-3cdc-4380-895e-1d2034642f3a admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:51.341159 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 714/1411] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:00:51 2022] GET /identity/v3/roles => generated 1420 bytes in 309 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:00:51.347135 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b508ded-3cdc-4380-895e-1d2034642f3a admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:51.398958 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b508ded-3cdc-4380-895e-1d2034642f3a admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:51.412959 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cabcbd7-47f7-4f87-82fb-1c9968bb1d22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:51.413892 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cabcbd7-47f7-4f87-82fb-1c9968bb1d22 admin admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:51.414025 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:00:51.414439 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cabcbd7-47f7-4f87-82fb-1c9968bb1d22 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:51.415032 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cabcbd7-47f7-4f87-82fb-1c9968bb1d22 admin admin] PATH_INFO: `/v3/projects/f36ee68417014697b9c0beb7f78cbb21/users/25b4b3a05a124c5587eadcb46bc041f4/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:51.418441 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cabcbd7-47f7-4f87-82fb-1c9968bb1d22 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f36ee68417014697b9c0beb7f78cbb21, user_id=25b4b3a05a124c5587eadcb46bc041f4, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:51.424883 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cabcbd7-47f7-4f87-82fb-1c9968bb1d22 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:51.448040 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 715/1412] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:00:51 2022] PUT /identity/v3/projects/f36ee68417014697b9c0beb7f78cbb21/users/25b4b3a05a124c5587eadcb46bc041f4/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:00:51.449474 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cabcbd7-47f7-4f87-82fb-1c9968bb1d22 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:51.451891 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-8cabcbd7-47f7-4f87-82fb-1c9968bb1d22 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:51.519966 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e23a90f7-3a0b-43f1-a2b5-3a8acdf4ef81 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:51.614427 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cabcbd7-47f7-4f87-82fb-1c9968bb1d22 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:51.624692 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-329b07a8-e232-4b41-8bf4-3d64a8727c69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:51.625832 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-329b07a8-e232-4b41-8bf4-3d64a8727c69 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:51.626220 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-329b07a8-e232-4b41-8bf4-3d64a8727c69 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:51.626635 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-329b07a8-e232-4b41-8bf4-3d64a8727c69 admin admin] PATH_INFO: `/v3/users/7db2cb676413491dade7fddb604cef46` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:51.628092 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-329b07a8-e232-4b41-8bf4-3d64a8727c69 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7db2cb676413491dade7fddb604cef46)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:51.630738 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-329b07a8-e232-4b41-8bf4-3d64a8727c69 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:51.675727 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-631af1f7-c828-455e-8ce4-f728291e6e56 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4d558918cd8b4ebea4fe8765297304db', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:51.676363 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-631af1f7-c828-455e-8ce4-f728291e6e56 service swift] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:51.676640 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-631af1f7-c828-455e-8ce4-f728291e6e56 service swift] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:51.676909 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-631af1f7-c828-455e-8ce4-f728291e6e56 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:51.749400 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-329b07a8-e232-4b41-8bf4-3d64a8727c69 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7db2cb676413491dade7fddb604cef46'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:51.761133 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-631af1f7-c828-455e-8ce4-f728291e6e56 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:51.763346 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-631af1f7-c828-455e-8ce4-f728291e6e56 service swift] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:51.764122 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-329b07a8-e232-4b41-8bf4-3d64a8727c69 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7db2cb676413491dade7fddb604cef46'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:51.775253 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 698/1413] 10.222.0.36 () {60 vars in 1384 bytes} [Thu Oct 20 09:00:51 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 575 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 20 09:00:51.776938 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b79b1cf-c7b6-4e01-ae50-ec41c8d42cdf None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:51.777228 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b79b1cf-c7b6-4e01-ae50-ec41c8d42cdf None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:51.777527 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b79b1cf-c7b6-4e01-ae50-ec41c8d42cdf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:51.802651 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-329b07a8-e232-4b41-8bf4-3d64a8727c69 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7db2cb676413491dade7fddb604cef46'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:51.816625 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-329b07a8-e232-4b41-8bf4-3d64a8727c69 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7db2cb676413491dade7fddb604cef46'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:51.817594 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-329b07a8-e232-4b41-8bf4-3d64a8727c69 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7db2cb676413491dade7fddb604cef46'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:51.825239 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 716/1414] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:00:51 2022] DELETE /identity/v3/users/7db2cb676413491dade7fddb604cef46 => generated 0 bytes in 376 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:00:51.826875 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33e23626-9c67-4aa9-bf4b-c198a2ea1094 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:51.827312 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33e23626-9c67-4aa9-bf4b-c198a2ea1094 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:51.827732 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33e23626-9c67-4aa9-bf4b-c198a2ea1094 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:51.843128 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-7b79b1cf-c7b6-4e01-ae50-ec41c8d42cdf None None] MFA Rules not processed for user `25b4b3a05a124c5587eadcb46bc041f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:00:51.846172 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-7b79b1cf-c7b6-4e01-ae50-ec41c8d42cdf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:51.851306 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-33e23626-9c67-4aa9-bf4b-c198a2ea1094 None None] MFA Rules not processed for user `8002ab26f6d64822aaa65aa730781b76`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:00:51.853261 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-33e23626-9c67-4aa9-bf4b-c198a2ea1094 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:51.923972 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:00:52.048951 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 717/1415] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:00:51 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 223 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:00:52.061867 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:00:52.067308 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e46edf6-7869-4cdf-b46f-719a9f1e6aa3 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:52.067735 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e46edf6-7869-4cdf-b46f-719a9f1e6aa3 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:52.069709 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e46edf6-7869-4cdf-b46f-719a9f1e6aa3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:52.102837 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 699/1416] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:00:51 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 325 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:00:52.115565 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b79b1cf-c7b6-4e01-ae50-ec41c8d42cdf None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:52.118129 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-7b79b1cf-c7b6-4e01-ae50-ec41c8d42cdf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:52.138459 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-3e46edf6-7869-4cdf-b46f-719a9f1e6aa3 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:00:52.140241 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-3e46edf6-7869-4cdf-b46f-719a9f1e6aa3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:52.315166 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 718/1417] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:00:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 262 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:00:52.366738 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e46edf6-7869-4cdf-b46f-719a9f1e6aa3 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:52.368830 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-3e46edf6-7869-4cdf-b46f-719a9f1e6aa3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:52.413947 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b79b1cf-c7b6-4e01-ae50-ec41c8d42cdf None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:52.447330 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-302534fe-ff71-4652-aded-b40fc83dadfe service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:52.447968 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-302534fe-ff71-4652-aded-b40fc83dadfe service cinder] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:52.448246 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-302534fe-ff71-4652-aded-b40fc83dadfe service cinder] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:52.448521 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-302534fe-ff71-4652-aded-b40fc83dadfe service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:52.485912 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-302534fe-ff71-4652-aded-b40fc83dadfe service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:52.491332 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-302534fe-ff71-4652-aded-b40fc83dadfe service cinder] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:52.513808 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 700/1418] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:00:52 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 400 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:00:52.519915 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e46edf6-7869-4cdf-b46f-719a9f1e6aa3 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:52.528623 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e057707c-a33a-445d-ab28-2c5bdcd417c2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:52.529506 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e057707c-a33a-445d-ab28-2c5bdcd417c2 service neutron] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:52.530705 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e057707c-a33a-445d-ab28-2c5bdcd417c2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:52.531254 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e057707c-a33a-445d-ab28-2c5bdcd417c2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:52.563325 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e057707c-a33a-445d-ab28-2c5bdcd417c2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:52.565359 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e057707c-a33a-445d-ab28-2c5bdcd417c2 service neutron] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:52.579048 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 719/1419] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:00:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 213 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 20 09:00:53.613051 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-302534fe-ff71-4652-aded-b40fc83dadfe service cinder] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:53.614787 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-302534fe-ff71-4652-aded-b40fc83dadfe service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:53.661271 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-302534fe-ff71-4652-aded-b40fc83dadfe service cinder] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:53.677054 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ccd9feb-9155-44ca-ac20-019b6ab727c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:53.679009 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ccd9feb-9155-44ca-ac20-019b6ab727c4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:53.680604 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ccd9feb-9155-44ca-ac20-019b6ab727c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:53.681697 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ccd9feb-9155-44ca-ac20-019b6ab727c4 admin admin] PATH_INFO: `/v3/users/a3bb6bab4d7646cca07529be996c86b8` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:53.684686 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ccd9feb-9155-44ca-ac20-019b6ab727c4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a3bb6bab4d7646cca07529be996c86b8)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:53.695045 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e057707c-a33a-445d-ab28-2c5bdcd417c2 service neutron] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:53.699175 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-e057707c-a33a-445d-ab28-2c5bdcd417c2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:53.710138 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ccd9feb-9155-44ca-ac20-019b6ab727c4 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:53.752734 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e057707c-a33a-445d-ab28-2c5bdcd417c2 service neutron] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:53.775379 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8527f37-91ac-44bd-88af-dbcd82c57809 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:53.779567 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8527f37-91ac-44bd-88af-dbcd82c57809 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:53.781547 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8527f37-91ac-44bd-88af-dbcd82c57809 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:53.782957 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8527f37-91ac-44bd-88af-dbcd82c57809 admin admin] PATH_INFO: `/v3/projects/ca0fdf3f133548f382e6072c8944538f` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:53.789135 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8527f37-91ac-44bd-88af-dbcd82c57809 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ca0fdf3f133548f382e6072c8944538f)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:53.799460 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8527f37-91ac-44bd-88af-dbcd82c57809 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:53.944690 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-1ccd9feb-9155-44ca-ac20-019b6ab727c4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a3bb6bab4d7646cca07529be996c86b8'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:53.977970 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-1ccd9feb-9155-44ca-ac20-019b6ab727c4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a3bb6bab4d7646cca07529be996c86b8'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:54.009680 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-c8527f37-91ac-44bd-88af-dbcd82c57809 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ca0fdf3f133548f382e6072c8944538f', 'request_id': 'req-c8527f37-91ac-44bd-88af-dbcd82c57809'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:54.027526 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-1ccd9feb-9155-44ca-ac20-019b6ab727c4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a3bb6bab4d7646cca07529be996c86b8'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:54.030040 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-c8527f37-91ac-44bd-88af-dbcd82c57809 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ca0fdf3f133548f382e6072c8944538f', 'request_id': 'req-c8527f37-91ac-44bd-88af-dbcd82c57809'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:54.040370 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-1ccd9feb-9155-44ca-ac20-019b6ab727c4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a3bb6bab4d7646cca07529be996c86b8'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:54.041704 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-1ccd9feb-9155-44ca-ac20-019b6ab727c4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a3bb6bab4d7646cca07529be996c86b8'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:54.052991 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 701/1420] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:00:53 2022] DELETE /identity/v3/users/a3bb6bab4d7646cca07529be996c86b8 => generated 0 bytes in 440 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:00:54.054743 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-c8527f37-91ac-44bd-88af-dbcd82c57809 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ca0fdf3f133548f382e6072c8944538f', 'request_id': 'req-c8527f37-91ac-44bd-88af-dbcd82c57809'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:54.057627 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e20cc4ce-d381-4ff9-a7ef-ecac82604e93 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:54.058891 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e20cc4ce-d381-4ff9-a7ef-ecac82604e93 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:54.059279 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e20cc4ce-d381-4ff9-a7ef-ecac82604e93 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:54.073218 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:00:54.087737 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-e20cc4ce-d381-4ff9-a7ef-ecac82604e93 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:00:54.089319 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-e20cc4ce-d381-4ff9-a7ef-ecac82604e93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:54.094011 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-c8527f37-91ac-44bd-88af-dbcd82c57809 admin admin] The token cache is being invalidate because project ca0fdf3f133548f382e6072c8944538f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81562) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:00:54.094356 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-c8527f37-91ac-44bd-88af-dbcd82c57809 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:54.113452 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-c8527f37-91ac-44bd-88af-dbcd82c57809 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:54.115306 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 720/1421] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:00:53 2022] DELETE /identity/v3/projects/ca0fdf3f133548f382e6072c8944538f => generated 0 bytes in 421 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:00:54.119737 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8527f37-91ac-44bd-88af-dbcd82c57809 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:54.121247 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-c8527f37-91ac-44bd-88af-dbcd82c57809 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:54.213235 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 702/1422] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:00:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:00:54.238835 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8527f37-91ac-44bd-88af-dbcd82c57809 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:54.245123 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e20cc4ce-d381-4ff9-a7ef-ecac82604e93 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:54.247479 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-e20cc4ce-d381-4ff9-a7ef-ecac82604e93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:54.262614 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:00:54.266431 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fb372ea-ef6d-4251-b4b9-50e3a35341af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:54.268627 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fb372ea-ef6d-4251-b4b9-50e3a35341af admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:54.269450 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fb372ea-ef6d-4251-b4b9-50e3a35341af admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:54.269578 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fb372ea-ef6d-4251-b4b9-50e3a35341af admin admin] PATH_INFO: `/v3/users/5969af5e5b8b4f849218478f86d3f631` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:54.272027 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fb372ea-ef6d-4251-b4b9-50e3a35341af admin admin] RBAC: Authorizing `identity:delete_user(user_id=5969af5e5b8b4f849218478f86d3f631)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:54.282267 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fb372ea-ef6d-4251-b4b9-50e3a35341af admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:54.439641 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e20cc4ce-d381-4ff9-a7ef-ecac82604e93 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:54.441462 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-5fb372ea-ef6d-4251-b4b9-50e3a35341af admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5969af5e5b8b4f849218478f86d3f631'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:54.463671 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc6c0f24-4e15-4252-8d5a-b6d752f88ecd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:54.466085 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6c0f24-4e15-4252-8d5a-b6d752f88ecd service neutron] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:54.466620 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6c0f24-4e15-4252-8d5a-b6d752f88ecd service neutron] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:54.467061 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6c0f24-4e15-4252-8d5a-b6d752f88ecd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:54.472619 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-5fb372ea-ef6d-4251-b4b9-50e3a35341af admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5969af5e5b8b4f849218478f86d3f631'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:54.498097 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc6c0f24-4e15-4252-8d5a-b6d752f88ecd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:54.502915 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc6c0f24-4e15-4252-8d5a-b6d752f88ecd service neutron] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:54.511069 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-5fb372ea-ef6d-4251-b4b9-50e3a35341af admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5969af5e5b8b4f849218478f86d3f631'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:54.538930 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-5fb372ea-ef6d-4251-b4b9-50e3a35341af admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5969af5e5b8b4f849218478f86d3f631'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:54.539717 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-5fb372ea-ef6d-4251-b4b9-50e3a35341af admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5969af5e5b8b4f849218478f86d3f631'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:54.543650 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 703/1423] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:00:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 299 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:00:54.558933 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 721/1424] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:00:54 2022] DELETE /identity/v3/users/5969af5e5b8b4f849218478f86d3f631 => generated 0 bytes in 435 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:00:54.937948 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c77d08-6bae-4d0a-93ea-06d8fac43d1a None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:54.938813 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c77d08-6bae-4d0a-93ea-06d8fac43d1a None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:54.939288 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c77d08-6bae-4d0a-93ea-06d8fac43d1a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:54.988450 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-67c77d08-6bae-4d0a-93ea-06d8fac43d1a None None] MFA Rules not processed for user `794ca389002d41748d7925216ed964a5`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:00:54.995116 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-67c77d08-6bae-4d0a-93ea-06d8fac43d1a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:55.006142 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fb372ea-ef6d-4251-b4b9-50e3a35341af admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:55.007997 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-5fb372ea-ef6d-4251-b4b9-50e3a35341af admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:55.244048 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fb372ea-ef6d-4251-b4b9-50e3a35341af admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:55.257606 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 704/1425] 10.222.0.23 () {58 vars in 1015 bytes} [Thu Oct 20 09:00:54 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 320 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:00:55.272474 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0bca4a8-0b52-4930-b12b-84045f52e032 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:55.273143 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0bca4a8-0b52-4930-b12b-84045f52e032 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:55.273426 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0bca4a8-0b52-4930-b12b-84045f52e032 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:55.273698 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0bca4a8-0b52-4930-b12b-84045f52e032 admin admin] PATH_INFO: `/v3/users/c8b47ebff9924b218344fb930f083ce1` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:55.288542 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0bca4a8-0b52-4930-b12b-84045f52e032 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c8b47ebff9924b218344fb930f083ce1)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:55.289102 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0bca4a8-0b52-4930-b12b-84045f52e032 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:55.385760 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:00:55.396666 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67c77d08-6bae-4d0a-93ea-06d8fac43d1a None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:55.398315 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-67c77d08-6bae-4d0a-93ea-06d8fac43d1a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:55.428484 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67c77d08-6bae-4d0a-93ea-06d8fac43d1a None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:55.443351 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44fdbd5f-e672-45b0-b1ae-e22cf7fc1550 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:55.471488 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44fdbd5f-e672-45b0-b1ae-e22cf7fc1550 service neutron] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:55.471928 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44fdbd5f-e672-45b0-b1ae-e22cf7fc1550 service neutron] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:55.472301 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44fdbd5f-e672-45b0-b1ae-e22cf7fc1550 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:55.512916 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44fdbd5f-e672-45b0-b1ae-e22cf7fc1550 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:55.521966 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44fdbd5f-e672-45b0-b1ae-e22cf7fc1550 service neutron] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:55.558335 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-b0bca4a8-0b52-4930-b12b-84045f52e032 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c8b47ebff9924b218344fb930f083ce1'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:55.571609 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 705/1426] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:00:55 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 176 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 20 09:00:55.587854 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-b0bca4a8-0b52-4930-b12b-84045f52e032 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c8b47ebff9924b218344fb930f083ce1'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:55.632249 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-b0bca4a8-0b52-4930-b12b-84045f52e032 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c8b47ebff9924b218344fb930f083ce1'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:55.647578 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-b0bca4a8-0b52-4930-b12b-84045f52e032 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c8b47ebff9924b218344fb930f083ce1'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:55.653319 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-b0bca4a8-0b52-4930-b12b-84045f52e032 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c8b47ebff9924b218344fb930f083ce1'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:55.668157 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 722/1427] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:00:55 2022] DELETE /identity/v3/users/c8b47ebff9924b218344fb930f083ce1 => generated 0 bytes in 662 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:00:55.675102 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8109ed62-11d6-4d62-b950-6ae0177d938c None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:55.676342 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8109ed62-11d6-4d62-b950-6ae0177d938c None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:55.677124 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8109ed62-11d6-4d62-b950-6ae0177d938c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:55.721674 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-8109ed62-11d6-4d62-b950-6ae0177d938c None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:00:55.745340 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-8109ed62-11d6-4d62-b950-6ae0177d938c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:55.963699 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0bca4a8-0b52-4930-b12b-84045f52e032 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:55.965884 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-b0bca4a8-0b52-4930-b12b-84045f52e032 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:56.136095 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 706/1428] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:00:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 461 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:00:56.162510 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8109ed62-11d6-4d62-b950-6ae0177d938c None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:56.166404 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-8109ed62-11d6-4d62-b950-6ae0177d938c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:56.306069 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0bca4a8-0b52-4930-b12b-84045f52e032 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:56.323957 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af1ef092-5834-43cc-885d-89f5315125fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:56.324825 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1ef092-5834-43cc-885d-89f5315125fe admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:56.325735 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1ef092-5834-43cc-885d-89f5315125fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:56.327438 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1ef092-5834-43cc-885d-89f5315125fe admin admin] PATH_INFO: `/v3/projects/a0542d0ca1eb4be69dd0b8b3ab26790a` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:56.331263 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af1ef092-5834-43cc-885d-89f5315125fe admin admin] RBAC: Authorizing `identity:delete_project(project_id=a0542d0ca1eb4be69dd0b8b3ab26790a)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:56.335171 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af1ef092-5834-43cc-885d-89f5315125fe admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:56.405953 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:00:56.418736 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8109ed62-11d6-4d62-b950-6ae0177d938c None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:56.449978 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2205738-15f7-4939-8160-9bad3e85a732 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:56.456072 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2205738-15f7-4939-8160-9bad3e85a732 service neutron] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:56.462079 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2205738-15f7-4939-8160-9bad3e85a732 service neutron] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:56.462600 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2205738-15f7-4939-8160-9bad3e85a732 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:56.528950 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2205738-15f7-4939-8160-9bad3e85a732 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:56.540974 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2205738-15f7-4939-8160-9bad3e85a732 service neutron] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:56.547616 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-af1ef092-5834-43cc-885d-89f5315125fe admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a0542d0ca1eb4be69dd0b8b3ab26790a', 'request_id': 'req-af1ef092-5834-43cc-885d-89f5315125fe'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:56.558865 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-af1ef092-5834-43cc-885d-89f5315125fe admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a0542d0ca1eb4be69dd0b8b3ab26790a', 'request_id': 'req-af1ef092-5834-43cc-885d-89f5315125fe'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:56.575796 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-af1ef092-5834-43cc-885d-89f5315125fe admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a0542d0ca1eb4be69dd0b8b3ab26790a', 'request_id': 'req-af1ef092-5834-43cc-885d-89f5315125fe'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:56.607247 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 707/1429] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:00:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 445 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:00:56.626597 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-af1ef092-5834-43cc-885d-89f5315125fe admin admin] The token cache is being invalidate because project a0542d0ca1eb4be69dd0b8b3ab26790a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81562) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:00:56.628413 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-af1ef092-5834-43cc-885d-89f5315125fe admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:56.632200 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-af1ef092-5834-43cc-885d-89f5315125fe admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:56.646615 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 723/1430] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:00:55 2022] DELETE /identity/v3/projects/a0542d0ca1eb4be69dd0b8b3ab26790a => generated 0 bytes in 681 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:00:56.647908 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2205738-15f7-4939-8160-9bad3e85a732 service neutron] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:56.668313 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2205738-15f7-4939-8160-9bad3e85a732 service neutron] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:56.683678 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af1ef092-5834-43cc-885d-89f5315125fe admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:56.685963 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-af1ef092-5834-43cc-885d-89f5315125fe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:56.779039 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:DELETE=16,keystone:SELECT=82,keystone:INSERT=2 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:00:56.929227 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f1f8432-5d54-498d-b570-2bc097e181c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:56.929939 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f1f8432-5d54-498d-b570-2bc097e181c3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:56.932373 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f1f8432-5d54-498d-b570-2bc097e181c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:56.932680 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f1f8432-5d54-498d-b570-2bc097e181c3 admin admin] PATH_INFO: `/v3/users/25b4b3a05a124c5587eadcb46bc041f4` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:56.935637 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f1f8432-5d54-498d-b570-2bc097e181c3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=25b4b3a05a124c5587eadcb46bc041f4)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:56.937427 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f1f8432-5d54-498d-b570-2bc097e181c3 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:56.995823 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af1ef092-5834-43cc-885d-89f5315125fe admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:57.013179 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cefc2f54-e03d-4c11-8433-2af984dc3b8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:57.014202 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cefc2f54-e03d-4c11-8433-2af984dc3b8d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:57.014647 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cefc2f54-e03d-4c11-8433-2af984dc3b8d admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:57.015549 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cefc2f54-e03d-4c11-8433-2af984dc3b8d admin admin] PATH_INFO: `/v3/projects/45f38f69246e473e88cb94e0cc264524` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:57.021307 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cefc2f54-e03d-4c11-8433-2af984dc3b8d admin admin] RBAC: Authorizing `identity:delete_project(project_id=45f38f69246e473e88cb94e0cc264524)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:57.024854 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cefc2f54-e03d-4c11-8433-2af984dc3b8d admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:57.157006 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-1f1f8432-5d54-498d-b570-2bc097e181c3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '25b4b3a05a124c5587eadcb46bc041f4'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:57.172498 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-cefc2f54-e03d-4c11-8433-2af984dc3b8d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '45f38f69246e473e88cb94e0cc264524', 'request_id': 'req-cefc2f54-e03d-4c11-8433-2af984dc3b8d'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:57.175619 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-1f1f8432-5d54-498d-b570-2bc097e181c3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '25b4b3a05a124c5587eadcb46bc041f4'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:57.200913 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-cefc2f54-e03d-4c11-8433-2af984dc3b8d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '45f38f69246e473e88cb94e0cc264524', 'request_id': 'req-cefc2f54-e03d-4c11-8433-2af984dc3b8d'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:57.221166 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-1f1f8432-5d54-498d-b570-2bc097e181c3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '25b4b3a05a124c5587eadcb46bc041f4'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:57.227674 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-cefc2f54-e03d-4c11-8433-2af984dc3b8d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '45f38f69246e473e88cb94e0cc264524', 'request_id': 'req-cefc2f54-e03d-4c11-8433-2af984dc3b8d'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:57.238586 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-1f1f8432-5d54-498d-b570-2bc097e181c3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '25b4b3a05a124c5587eadcb46bc041f4'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:57.239518 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-1f1f8432-5d54-498d-b570-2bc097e181c3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '25b4b3a05a124c5587eadcb46bc041f4'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:57.254283 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 708/1431] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:00:56 2022] DELETE /identity/v3/users/25b4b3a05a124c5587eadcb46bc041f4 => generated 0 bytes in 607 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:00:57.259609 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-cefc2f54-e03d-4c11-8433-2af984dc3b8d admin admin] The token cache is being invalidate because project 45f38f69246e473e88cb94e0cc264524 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81562) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:00:57.259907 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-cefc2f54-e03d-4c11-8433-2af984dc3b8d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:57.260958 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-cefc2f54-e03d-4c11-8433-2af984dc3b8d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:57.262877 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 724/1432] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:00:56 2022] DELETE /identity/v3/projects/45f38f69246e473e88cb94e0cc264524 => generated 0 bytes in 580 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:00:57.268303 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e285d71-82fc-42bd-9db4-9672c46c022e None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:57.268586 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e285d71-82fc-42bd-9db4-9672c46c022e None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:57.268860 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e285d71-82fc-42bd-9db4-9672c46c022e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:57.306266 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-2e285d71-82fc-42bd-9db4-9672c46c022e None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:00:57.307631 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-2e285d71-82fc-42bd-9db4-9672c46c022e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:57.500939 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 709/1433] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:00:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 234 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:00:57.504133 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e89b23d-1273-47ef-9022-062dd1ee8d4a None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:57.504826 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e89b23d-1273-47ef-9022-062dd1ee8d4a None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:57.505278 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e89b23d-1273-47ef-9022-062dd1ee8d4a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:57.616274 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-7e89b23d-1273-47ef-9022-062dd1ee8d4a None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:00:57.619584 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-7e89b23d-1273-47ef-9022-062dd1ee8d4a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:57.643848 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 725/1434] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:00:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:00:57.647731 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e285d71-82fc-42bd-9db4-9672c46c022e None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:57.665283 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e285d71-82fc-42bd-9db4-9672c46c022e None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:57.709800 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95964093-08e5-41e1-9c30-f20adeee2c1f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:57.710555 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95964093-08e5-41e1-9c30-f20adeee2c1f admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:57.710830 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95964093-08e5-41e1-9c30-f20adeee2c1f admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:57.711137 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95964093-08e5-41e1-9c30-f20adeee2c1f admin admin] PATH_INFO: `/v3/domains` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:57.712570 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95964093-08e5-41e1-9c30-f20adeee2c1f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 09:00:57.712871 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95964093-08e5-41e1-9c30-f20adeee2c1f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:57.722153 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95964093-08e5-41e1-9c30-f20adeee2c1f admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:57.752378 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:00:57.774868 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e89b23d-1273-47ef-9022-062dd1ee8d4a None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:57.779316 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 710/1435] 10.222.0.36 () {58 vars in 1153 bytes} [Thu Oct 20 09:00:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 132 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 09:00:57.787587 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e89b23d-1273-47ef-9022-062dd1ee8d4a None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:57.807704 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d47014a-0f2a-4dc4-9ea0-281fda22ea7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:57.808434 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d47014a-0f2a-4dc4-9ea0-281fda22ea7f admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:57.809648 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d47014a-0f2a-4dc4-9ea0-281fda22ea7f admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:57.809903 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d47014a-0f2a-4dc4-9ea0-281fda22ea7f admin admin] PATH_INFO: `/v3/projects` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:57.810598 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d47014a-0f2a-4dc4-9ea0-281fda22ea7f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:57.812589 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d47014a-0f2a-4dc4-9ea0-281fda22ea7f admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:57.880390 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95964093-08e5-41e1-9c30-f20adeee2c1f admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:57.880755 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 726/1436] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:00:57 2022] POST /identity/v3/projects => generated 365 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 20 09:00:57.895682 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95964093-08e5-41e1-9c30-f20adeee2c1f admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:58.069279 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e43c0fd0-842c-4832-9909-10c0c1a3cd85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:58.069975 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e43c0fd0-842c-4832-9909-10c0c1a3cd85 admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:58.070159 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e43c0fd0-842c-4832-9909-10c0c1a3cd85 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:58.070432 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e43c0fd0-842c-4832-9909-10c0c1a3cd85 admin admin] PATH_INFO: `/v3/users` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:58.071806 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e43c0fd0-842c-4832-9909-10c0c1a3cd85 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:58.073940 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e43c0fd0-842c-4832-9909-10c0c1a3cd85 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:58.131079 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 711/1437] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:00:57 2022] POST /identity/v3/users => generated 283 bytes in 251 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 20 09:00:58.137131 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d47014a-0f2a-4dc4-9ea0-281fda22ea7f admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:58.153094 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d47014a-0f2a-4dc4-9ea0-281fda22ea7f admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:58.199954 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff54914-a324-4fdf-a60c-37bb49128596 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:58.205439 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff54914-a324-4fdf-a60c-37bb49128596 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:58.207812 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff54914-a324-4fdf-a60c-37bb49128596 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:58.208261 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff54914-a324-4fdf-a60c-37bb49128596 admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:58.209091 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bff54914-a324-4fdf-a60c-37bb49128596 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:58.216347 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bff54914-a324-4fdf-a60c-37bb49128596 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:58.268181 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 727/1438] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:00:58 2022] GET /identity/v3/roles => generated 1420 bytes in 132 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:00:58.272362 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e43c0fd0-842c-4832-9909-10c0c1a3cd85 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:58.284974 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e43c0fd0-842c-4832-9909-10c0c1a3cd85 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:58.299240 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94da6a52-2e20-4f7f-b156-1753f5e39c7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:58.299874 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94da6a52-2e20-4f7f-b156-1753f5e39c7e admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:58.300145 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94da6a52-2e20-4f7f-b156-1753f5e39c7e admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:58.300413 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94da6a52-2e20-4f7f-b156-1753f5e39c7e admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:58.301636 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94da6a52-2e20-4f7f-b156-1753f5e39c7e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:58.303068 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94da6a52-2e20-4f7f-b156-1753f5e39c7e admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:58.324855 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 712/1439] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:00:58 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:00:58.329124 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff54914-a324-4fdf-a60c-37bb49128596 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:58.351418 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff54914-a324-4fdf-a60c-37bb49128596 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:58.391726 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1497cd96-aeaa-4488-976f-98ccf3a1ce49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:58.392659 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1497cd96-aeaa-4488-976f-98ccf3a1ce49 admin admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:58.394564 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1497cd96-aeaa-4488-976f-98ccf3a1ce49 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:58.395188 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1497cd96-aeaa-4488-976f-98ccf3a1ce49 admin admin] PATH_INFO: `/v3/projects/379e05649b904064be9a1055ba59b895/users/874a9c0c0ff7445b9465e67847c3b4c0/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:58.449734 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1497cd96-aeaa-4488-976f-98ccf3a1ce49 admin admin] RBAC: Authorizing `identity:create_grant(project_id=379e05649b904064be9a1055ba59b895, user_id=874a9c0c0ff7445b9465e67847c3b4c0, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:58.455685 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1497cd96-aeaa-4488-976f-98ccf3a1ce49 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:58.530079 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 728/1440] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:00:58 2022] PUT /identity/v3/projects/379e05649b904064be9a1055ba59b895/users/874a9c0c0ff7445b9465e67847c3b4c0/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:00:58.533331 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3957f83e-0904-4c46-9c12-624d5eb82fb4 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:58.533832 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3957f83e-0904-4c46-9c12-624d5eb82fb4 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:58.534384 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3957f83e-0904-4c46-9c12-624d5eb82fb4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:58.664982 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-3957f83e-0904-4c46-9c12-624d5eb82fb4 None None] MFA Rules not processed for user `874a9c0c0ff7445b9465e67847c3b4c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:00:58.666252 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-3957f83e-0904-4c46-9c12-624d5eb82fb4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:58.677459 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1497cd96-aeaa-4488-976f-98ccf3a1ce49 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:58.680583 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-1497cd96-aeaa-4488-976f-98ccf3a1ce49 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:58.900173 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:00:58.935474 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 713/1441] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:00:58 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 403 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:00:58.939719 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1497cd96-aeaa-4488-976f-98ccf3a1ce49 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:58.939865 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3957f83e-0904-4c46-9c12-624d5eb82fb4 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:58.947366 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c31256a-92f6-478a-a600-8856ea70e881 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:58.947990 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c31256a-92f6-478a-a600-8856ea70e881 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:58.948271 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c31256a-92f6-478a-a600-8856ea70e881 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:58.948537 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c31256a-92f6-478a-a600-8856ea70e881 admin admin] PATH_INFO: `/v3/projects/fead5997db8244ac9b7cc96e21eb7178` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:58.949672 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c31256a-92f6-478a-a600-8856ea70e881 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fead5997db8244ac9b7cc96e21eb7178)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:58.952000 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c31256a-92f6-478a-a600-8856ea70e881 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:58.982637 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3957f83e-0904-4c46-9c12-624d5eb82fb4 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:59.001495 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d64f55-ac43-4439-a28f-e91caec18dbe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:59.002638 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d64f55-ac43-4439-a28f-e91caec18dbe admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:59.003067 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d64f55-ac43-4439-a28f-e91caec18dbe admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:59.003510 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d64f55-ac43-4439-a28f-e91caec18dbe admin admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:59.004365 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8d64f55-ac43-4439-a28f-e91caec18dbe admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:59.006712 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8d64f55-ac43-4439-a28f-e91caec18dbe admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:59.125785 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 714/1442] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:00:58 2022] POST /identity/v3/projects => generated 363 bytes in 186 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:00:59.131686 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d64f55-ac43-4439-a28f-e91caec18dbe admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:59.140018 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-c8d64f55-ac43-4439-a28f-e91caec18dbe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:59.144131 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-5c31256a-92f6-478a-a600-8856ea70e881 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fead5997db8244ac9b7cc96e21eb7178', 'request_id': 'req-5c31256a-92f6-478a-a600-8856ea70e881'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:59.179020 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-5c31256a-92f6-478a-a600-8856ea70e881 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fead5997db8244ac9b7cc96e21eb7178', 'request_id': 'req-5c31256a-92f6-478a-a600-8856ea70e881'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:59.196131 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-5c31256a-92f6-478a-a600-8856ea70e881 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fead5997db8244ac9b7cc96e21eb7178', 'request_id': 'req-5c31256a-92f6-478a-a600-8856ea70e881'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:59.229539 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-5c31256a-92f6-478a-a600-8856ea70e881 admin admin] The token cache is being invalidate because project fead5997db8244ac9b7cc96e21eb7178 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81562) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:00:59.229999 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-5c31256a-92f6-478a-a600-8856ea70e881 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:59.230903 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-5c31256a-92f6-478a-a600-8856ea70e881 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:59.239345 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 729/1443] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:00:58 2022] DELETE /identity/v3/projects/fead5997db8244ac9b7cc96e21eb7178 => generated 0 bytes in 562 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:00:59.245528 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c31256a-92f6-478a-a600-8856ea70e881 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:59.251551 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-5c31256a-92f6-478a-a600-8856ea70e881 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:59.347453 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:00:59.393029 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d64f55-ac43-4439-a28f-e91caec18dbe admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:59.401472 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59aaaeb3-6a49-4d39-af8a-efd33213c39c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:59.402788 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59aaaeb3-6a49-4d39-af8a-efd33213c39c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:59.403352 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59aaaeb3-6a49-4d39-af8a-efd33213c39c admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:59.403766 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59aaaeb3-6a49-4d39-af8a-efd33213c39c admin admin] PATH_INFO: `/v3/projects/cb4e66b2ef2b4d79b6b18a0e0c801719` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:59.407975 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59aaaeb3-6a49-4d39-af8a-efd33213c39c admin admin] RBAC: Authorizing `identity:delete_project(project_id=cb4e66b2ef2b4d79b6b18a0e0c801719)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:59.411437 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59aaaeb3-6a49-4d39-af8a-efd33213c39c admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:59.519271 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c31256a-92f6-478a-a600-8856ea70e881 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:59.583973 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-59aaaeb3-6a49-4d39-af8a-efd33213c39c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cb4e66b2ef2b4d79b6b18a0e0c801719', 'request_id': 'req-59aaaeb3-6a49-4d39-af8a-efd33213c39c'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:59.604866 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-59aaaeb3-6a49-4d39-af8a-efd33213c39c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cb4e66b2ef2b4d79b6b18a0e0c801719', 'request_id': 'req-59aaaeb3-6a49-4d39-af8a-efd33213c39c'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:59.622714 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-59aaaeb3-6a49-4d39-af8a-efd33213c39c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cb4e66b2ef2b4d79b6b18a0e0c801719', 'request_id': 'req-59aaaeb3-6a49-4d39-af8a-efd33213c39c'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:59.669017 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-59aaaeb3-6a49-4d39-af8a-efd33213c39c admin admin] The token cache is being invalidate because project cb4e66b2ef2b4d79b6b18a0e0c801719 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81561) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:00:59.669320 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-59aaaeb3-6a49-4d39-af8a-efd33213c39c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:59.671778 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-59aaaeb3-6a49-4d39-af8a-efd33213c39c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:00:59.674130 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 715/1444] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:00:59 2022] DELETE /identity/v3/projects/cb4e66b2ef2b4d79b6b18a0e0c801719 => generated 0 bytes in 543 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:00:59.674957 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59aaaeb3-6a49-4d39-af8a-efd33213c39c admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:59.682725 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-59aaaeb3-6a49-4d39-af8a-efd33213c39c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:59.712414 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c4f5737-9c38-4394-b81a-3a3b30af42c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:59.716668 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4f5737-9c38-4394-b81a-3a3b30af42c3 admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:59.716668 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4f5737-9c38-4394-b81a-3a3b30af42c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:59.716880 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4f5737-9c38-4394-b81a-3a3b30af42c3 admin admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:59.717574 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c4f5737-9c38-4394-b81a-3a3b30af42c3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:59.719697 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c4f5737-9c38-4394-b81a-3a3b30af42c3 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:59.812691 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 730/1445] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:00:59 2022] POST /identity/v3/users => generated 288 bytes in 567 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:00:59.812691 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c4f5737-9c38-4394-b81a-3a3b30af42c3 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:00:59.814310 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-8c4f5737-9c38-4394-b81a-3a3b30af42c3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:00:59.849671 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c4f5737-9c38-4394-b81a-3a3b30af42c3 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:59.869667 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e7ab562-65e9-4abb-a99b-c8aa6f53a26f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:59.870363 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e7ab562-65e9-4abb-a99b-c8aa6f53a26f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:59.870796 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e7ab562-65e9-4abb-a99b-c8aa6f53a26f admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:59.871175 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e7ab562-65e9-4abb-a99b-c8aa6f53a26f admin admin] PATH_INFO: `/v3/users/9bdafc0e45434e5c954518259493df5d` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:59.879274 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e7ab562-65e9-4abb-a99b-c8aa6f53a26f admin admin] RBAC: Authorizing `identity:delete_user(user_id=9bdafc0e45434e5c954518259493df5d)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:59.891349 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e7ab562-65e9-4abb-a99b-c8aa6f53a26f admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:59.917217 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=88,keystone:DELETE=8,keystone:INSERT=4 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:00:59.958094 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59aaaeb3-6a49-4d39-af8a-efd33213c39c admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:00:59.978279 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65de8694-a8a4-42c6-a366-ea7f0b7498ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:00:59.980395 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65de8694-a8a4-42c6-a366-ea7f0b7498ad admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:00:59.982787 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65de8694-a8a4-42c6-a366-ea7f0b7498ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:00:59.984429 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65de8694-a8a4-42c6-a366-ea7f0b7498ad admin admin] PATH_INFO: `/v3/users/ab7b646274c34e4089ebe5ccd7511ed3` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:00:59.987490 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65de8694-a8a4-42c6-a366-ea7f0b7498ad admin admin] RBAC: Authorizing `identity:delete_user(user_id=ab7b646274c34e4089ebe5ccd7511ed3)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:00:59.993336 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65de8694-a8a4-42c6-a366-ea7f0b7498ad admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:00:59.994250 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-6e7ab562-65e9-4abb-a99b-c8aa6f53a26f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9bdafc0e45434e5c954518259493df5d'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:00.016821 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-6e7ab562-65e9-4abb-a99b-c8aa6f53a26f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9bdafc0e45434e5c954518259493df5d'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:00.049449 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-6e7ab562-65e9-4abb-a99b-c8aa6f53a26f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9bdafc0e45434e5c954518259493df5d'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:00.068872 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-6e7ab562-65e9-4abb-a99b-c8aa6f53a26f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9bdafc0e45434e5c954518259493df5d'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:00.070194 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-6e7ab562-65e9-4abb-a99b-c8aa6f53a26f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9bdafc0e45434e5c954518259493df5d'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:00.104634 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 731/1446] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:00:59 2022] DELETE /identity/v3/users/9bdafc0e45434e5c954518259493df5d => generated 0 bytes in 296 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:00.105534 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e7ab562-65e9-4abb-a99b-c8aa6f53a26f admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:00.111201 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-6e7ab562-65e9-4abb-a99b-c8aa6f53a26f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:00.120921 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-65de8694-a8a4-42c6-a366-ea7f0b7498ad admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ab7b646274c34e4089ebe5ccd7511ed3'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:00.151404 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-65de8694-a8a4-42c6-a366-ea7f0b7498ad admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ab7b646274c34e4089ebe5ccd7511ed3'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:00.210016 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-65de8694-a8a4-42c6-a366-ea7f0b7498ad admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ab7b646274c34e4089ebe5ccd7511ed3'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:00.229762 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:00.242270 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-65de8694-a8a4-42c6-a366-ea7f0b7498ad admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ab7b646274c34e4089ebe5ccd7511ed3'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:00.247447 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-65de8694-a8a4-42c6-a366-ea7f0b7498ad admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ab7b646274c34e4089ebe5ccd7511ed3'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:00.269475 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 716/1447] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:00:59 2022] DELETE /identity/v3/users/ab7b646274c34e4089ebe5ccd7511ed3 => generated 0 bytes in 592 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:00.281455 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e7ab562-65e9-4abb-a99b-c8aa6f53a26f admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:00.438892 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b24325ae-36a2-44c7-b1fd-b09419e9b873 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:00.439792 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24325ae-36a2-44c7-b1fd-b09419e9b873 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:00.440219 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24325ae-36a2-44c7-b1fd-b09419e9b873 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:00.440638 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24325ae-36a2-44c7-b1fd-b09419e9b873 admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:00.441775 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b24325ae-36a2-44c7-b1fd-b09419e9b873 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:00.456569 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b24325ae-36a2-44c7-b1fd-b09419e9b873 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:00.472177 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 732/1448] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:01:00 2022] GET /identity/v3/roles => generated 1420 bytes in 367 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:01:00.477440 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65de8694-a8a4-42c6-a366-ea7f0b7498ad admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:00.479308 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-65de8694-a8a4-42c6-a366-ea7f0b7498ad admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:00.505124 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65de8694-a8a4-42c6-a366-ea7f0b7498ad admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:00.519532 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3436fc0f-50f2-429f-b58b-6335d230ee20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:00.523226 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3436fc0f-50f2-429f-b58b-6335d230ee20 admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:00.523705 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3436fc0f-50f2-429f-b58b-6335d230ee20 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:00.524092 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3436fc0f-50f2-429f-b58b-6335d230ee20 admin admin] PATH_INFO: `/v3/projects/a7907859f9e7402bbdd9921a221ed75c/users/10e0cfe4fbfc417c9ec4a7038e5efe5b/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:00.562600 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3436fc0f-50f2-429f-b58b-6335d230ee20 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a7907859f9e7402bbdd9921a221ed75c, user_id=10e0cfe4fbfc417c9ec4a7038e5efe5b, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:00.568548 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3436fc0f-50f2-429f-b58b-6335d230ee20 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:00.596335 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 717/1449] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:01:00 2022] PUT /identity/v3/projects/a7907859f9e7402bbdd9921a221ed75c/users/10e0cfe4fbfc417c9ec4a7038e5efe5b/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 120 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:00.600405 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b24325ae-36a2-44c7-b1fd-b09419e9b873 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:00.624238 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b24325ae-36a2-44c7-b1fd-b09419e9b873 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:00.747881 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:00.826939 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-567b804c-17fd-4750-bab3-831581f09b77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:00.828357 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567b804c-17fd-4750-bab3-831581f09b77 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:00.829364 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567b804c-17fd-4750-bab3-831581f09b77 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:00.830430 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567b804c-17fd-4750-bab3-831581f09b77 admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:00.833481 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-567b804c-17fd-4750-bab3-831581f09b77 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:00.836620 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-567b804c-17fd-4750-bab3-831581f09b77 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:00.894633 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 733/1450] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:01:00 2022] GET /identity/v3/roles => generated 1420 bytes in 295 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:01:00.900120 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3436fc0f-50f2-429f-b58b-6335d230ee20 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:00.923446 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3436fc0f-50f2-429f-b58b-6335d230ee20 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:00.944582 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb0a1129-4a48-45d5-afe5-9f890cb19a49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:00.945204 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb0a1129-4a48-45d5-afe5-9f890cb19a49 admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:00.945511 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb0a1129-4a48-45d5-afe5-9f890cb19a49 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:00.945787 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb0a1129-4a48-45d5-afe5-9f890cb19a49 admin admin] PATH_INFO: `/v3/projects/a7907859f9e7402bbdd9921a221ed75c/users/10e0cfe4fbfc417c9ec4a7038e5efe5b/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:00.952383 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb0a1129-4a48-45d5-afe5-9f890cb19a49 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a7907859f9e7402bbdd9921a221ed75c, user_id=10e0cfe4fbfc417c9ec4a7038e5efe5b, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:00.959840 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb0a1129-4a48-45d5-afe5-9f890cb19a49 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:00.990095 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 718/1451] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:01:00 2022] PUT /identity/v3/projects/a7907859f9e7402bbdd9921a221ed75c/users/10e0cfe4fbfc417c9ec4a7038e5efe5b/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 91 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:01.001642 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd61783-01a3-4df3-b989-7c0e85ade402 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:01.001990 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd61783-01a3-4df3-b989-7c0e85ade402 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:01.002360 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd61783-01a3-4df3-b989-7c0e85ade402 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:01.087503 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-4fd61783-01a3-4df3-b989-7c0e85ade402 None None] MFA Rules not processed for user `10e0cfe4fbfc417c9ec4a7038e5efe5b`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:01.089137 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-4fd61783-01a3-4df3-b989-7c0e85ade402 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:01.248655 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 734/1452] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:00 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 248 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:01.260632 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb0a1129-4a48-45d5-afe5-9f890cb19a49 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:01.263033 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-eb0a1129-4a48-45d5-afe5-9f890cb19a49 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:01.391638 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb0a1129-4a48-45d5-afe5-9f890cb19a49 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:01.396491 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48ec3dfa-306c-4816-a66a-88728c6d8cf6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:01.397257 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48ec3dfa-306c-4816-a66a-88728c6d8cf6 service cinder] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:01.397653 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48ec3dfa-306c-4816-a66a-88728c6d8cf6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:01.398246 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48ec3dfa-306c-4816-a66a-88728c6d8cf6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:01.410578 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48ec3dfa-306c-4816-a66a-88728c6d8cf6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:01.412322 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48ec3dfa-306c-4816-a66a-88728c6d8cf6 service cinder] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:01.436029 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 719/1453] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:01:01 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 166 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:01:01.559486 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fd61783-01a3-4df3-b989-7c0e85ade402 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:01.576087 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48ec3dfa-306c-4816-a66a-88728c6d8cf6 service cinder] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:01.578197 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-48ec3dfa-306c-4816-a66a-88728c6d8cf6 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:01.583374 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fd61783-01a3-4df3-b989-7c0e85ade402 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:01.595200 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd9b698b-c862-45fa-bc76-b60b7f46b0eb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:01.595842 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd9b698b-c862-45fa-bc76-b60b7f46b0eb service cinder] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:01.596124 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd9b698b-c862-45fa-bc76-b60b7f46b0eb service cinder] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:01.596396 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd9b698b-c862-45fa-bc76-b60b7f46b0eb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:01.602746 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-fd9b698b-c862-45fa-bc76-b60b7f46b0eb service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:01.744769 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48ec3dfa-306c-4816-a66a-88728c6d8cf6 service cinder] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:01.765902 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e48b27cf-fa89-4203-b3c8-bed5f8b5ceaa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:01.766995 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48b27cf-fa89-4203-b3c8-bed5f8b5ceaa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:01.767357 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48b27cf-fa89-4203-b3c8-bed5f8b5ceaa admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:01.767804 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48b27cf-fa89-4203-b3c8-bed5f8b5ceaa admin admin] PATH_INFO: `/v3/projects/f36ee68417014697b9c0beb7f78cbb21` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:01.770100 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e48b27cf-fa89-4203-b3c8-bed5f8b5ceaa admin admin] RBAC: Authorizing `identity:delete_project(project_id=f36ee68417014697b9c0beb7f78cbb21)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:01.772857 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e48b27cf-fa89-4203-b3c8-bed5f8b5ceaa admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:01.809877 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd9b698b-c862-45fa-bc76-b60b7f46b0eb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:01.812496 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd9b698b-c862-45fa-bc76-b60b7f46b0eb service cinder] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:01.843087 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 735/1454] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:01:01 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 284 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:01:01.869656 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd9b698b-c862-45fa-bc76-b60b7f46b0eb service cinder] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:01.871711 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-fd9b698b-c862-45fa-bc76-b60b7f46b0eb service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:01.904297 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd9b698b-c862-45fa-bc76-b60b7f46b0eb service cinder] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:01.961823 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-e48b27cf-fa89-4203-b3c8-bed5f8b5ceaa admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f36ee68417014697b9c0beb7f78cbb21', 'request_id': 'req-e48b27cf-fa89-4203-b3c8-bed5f8b5ceaa'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:01.975277 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-e48b27cf-fa89-4203-b3c8-bed5f8b5ceaa admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f36ee68417014697b9c0beb7f78cbb21', 'request_id': 'req-e48b27cf-fa89-4203-b3c8-bed5f8b5ceaa'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:02.003475 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=92,keystone:INSERT=2,keystone:DELETE=6 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:02.011208 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-e48b27cf-fa89-4203-b3c8-bed5f8b5ceaa admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f36ee68417014697b9c0beb7f78cbb21', 'request_id': 'req-e48b27cf-fa89-4203-b3c8-bed5f8b5ceaa'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:02.067581 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-e48b27cf-fa89-4203-b3c8-bed5f8b5ceaa admin admin] The token cache is being invalidate because project f36ee68417014697b9c0beb7f78cbb21 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81561) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:01:02.070313 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-e48b27cf-fa89-4203-b3c8-bed5f8b5ceaa admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:02.074197 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-e48b27cf-fa89-4203-b3c8-bed5f8b5ceaa admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:02.075269 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:02.082380 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 720/1455] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:01:01 2022] DELETE /identity/v3/projects/f36ee68417014697b9c0beb7f78cbb21 => generated 0 bytes in 507 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:01:02.088147 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e48b27cf-fa89-4203-b3c8-bed5f8b5ceaa admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:02.092756 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-e48b27cf-fa89-4203-b3c8-bed5f8b5ceaa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:02.093686 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c520eae8-a1fd-4c7f-a45b-99813babad56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:02.094335 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c520eae8-a1fd-4c7f-a45b-99813babad56 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:02.094644 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c520eae8-a1fd-4c7f-a45b-99813babad56 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:02.094914 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c520eae8-a1fd-4c7f-a45b-99813babad56 admin admin] PATH_INFO: `/v3/projects/49357129667f4f6581ec7b9dbb6c8fbc` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:02.097721 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c520eae8-a1fd-4c7f-a45b-99813babad56 admin admin] RBAC: Authorizing `identity:delete_project(project_id=49357129667f4f6581ec7b9dbb6c8fbc)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:02.099722 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c520eae8-a1fd-4c7f-a45b-99813babad56 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:02.141468 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e48b27cf-fa89-4203-b3c8-bed5f8b5ceaa admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:02.156214 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce4098f-d66d-4026-8bea-8519caba2767 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:02.157998 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce4098f-d66d-4026-8bea-8519caba2767 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:02.158443 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce4098f-d66d-4026-8bea-8519caba2767 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:02.159126 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce4098f-d66d-4026-8bea-8519caba2767 admin admin] PATH_INFO: `/v3/projects/c2b7484f3ae34596bae4794f348cd875` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:02.160546 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ce4098f-d66d-4026-8bea-8519caba2767 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c2b7484f3ae34596bae4794f348cd875)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:02.162909 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ce4098f-d66d-4026-8bea-8519caba2767 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:02.215217 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-c520eae8-a1fd-4c7f-a45b-99813babad56 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '49357129667f4f6581ec7b9dbb6c8fbc', 'request_id': 'req-c520eae8-a1fd-4c7f-a45b-99813babad56'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:02.234876 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-c520eae8-a1fd-4c7f-a45b-99813babad56 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '49357129667f4f6581ec7b9dbb6c8fbc', 'request_id': 'req-c520eae8-a1fd-4c7f-a45b-99813babad56'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:02.265847 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-c520eae8-a1fd-4c7f-a45b-99813babad56 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '49357129667f4f6581ec7b9dbb6c8fbc', 'request_id': 'req-c520eae8-a1fd-4c7f-a45b-99813babad56'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:02.314406 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-3ce4098f-d66d-4026-8bea-8519caba2767 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c2b7484f3ae34596bae4794f348cd875', 'request_id': 'req-3ce4098f-d66d-4026-8bea-8519caba2767'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:02.315699 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-c520eae8-a1fd-4c7f-a45b-99813babad56 admin admin] The token cache is being invalidate because project 49357129667f4f6581ec7b9dbb6c8fbc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81562) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:01:02.316441 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-c520eae8-a1fd-4c7f-a45b-99813babad56 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:02.319029 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-c520eae8-a1fd-4c7f-a45b-99813babad56 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:02.325172 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 736/1456] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:01:01 2022] DELETE /identity/v3/projects/49357129667f4f6581ec7b9dbb6c8fbc => generated 0 bytes in 455 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:02.325172 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe31a83-da8a-48ed-a68a-81abd5d0d712 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:02.325172 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe31a83-da8a-48ed-a68a-81abd5d0d712 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:02.325172 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe31a83-da8a-48ed-a68a-81abd5d0d712 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:02.335520 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-3ce4098f-d66d-4026-8bea-8519caba2767 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c2b7484f3ae34596bae4794f348cd875', 'request_id': 'req-3ce4098f-d66d-4026-8bea-8519caba2767'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:02.349126 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-3ce4098f-d66d-4026-8bea-8519caba2767 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c2b7484f3ae34596bae4794f348cd875', 'request_id': 'req-3ce4098f-d66d-4026-8bea-8519caba2767'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:02.372697 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-abe31a83-da8a-48ed-a68a-81abd5d0d712 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:02.374980 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-abe31a83-da8a-48ed-a68a-81abd5d0d712 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:02.395846 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-3ce4098f-d66d-4026-8bea-8519caba2767 admin admin] The token cache is being invalidate because project c2b7484f3ae34596bae4794f348cd875 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81561) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:01:02.396146 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-3ce4098f-d66d-4026-8bea-8519caba2767 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:02.398163 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-3ce4098f-d66d-4026-8bea-8519caba2767 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:02.407623 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 721/1457] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:01:02 2022] DELETE /identity/v3/projects/c2b7484f3ae34596bae4794f348cd875 => generated 0 bytes in 322 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:02.408578 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ad0923-fc59-4cad-99f3-686d3a7fbef7 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:02.409689 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ad0923-fc59-4cad-99f3-686d3a7fbef7 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:02.410994 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ad0923-fc59-4cad-99f3-686d3a7fbef7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:02.447397 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-b1ad0923-fc59-4cad-99f3-686d3a7fbef7 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:02.448816 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-b1ad0923-fc59-4cad-99f3-686d3a7fbef7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:02.586915 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 722/1458] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:02.588789 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dafecdc7-5572-4b0d-852d-c0feceb9274e None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:02.589083 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dafecdc7-5572-4b0d-852d-c0feceb9274e None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:02.589535 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dafecdc7-5572-4b0d-852d-c0feceb9274e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:02.590616 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 737/1459] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 267 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:02.592471 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-737bac5f-099d-4a10-baeb-cc3fec9de627 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:02.592758 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-737bac5f-099d-4a10-baeb-cc3fec9de627 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:02.593045 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-737bac5f-099d-4a10-baeb-cc3fec9de627 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:02.622502 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-dafecdc7-5572-4b0d-852d-c0feceb9274e None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:02.624452 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-dafecdc7-5572-4b0d-852d-c0feceb9274e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:02.655992 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 723/1460] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 69 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:02.661990 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e7225d-7045-4592-a97f-f122faf0e823 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:02.661990 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e7225d-7045-4592-a97f-f122faf0e823 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:02.662251 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-737bac5f-099d-4a10-baeb-cc3fec9de627 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:02.662327 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e7225d-7045-4592-a97f-f122faf0e823 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:02.663476 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-737bac5f-099d-4a10-baeb-cc3fec9de627 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:02.689905 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-38e7225d-7045-4592-a97f-f122faf0e823 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:02.696981 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 738/1461] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:02.698068 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f301b4-119d-4506-8412-512109f1b4a5 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:02.698348 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f301b4-119d-4506-8412-512109f1b4a5 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:02.698646 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f301b4-119d-4506-8412-512109f1b4a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:02.704226 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-38e7225d-7045-4592-a97f-f122faf0e823 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:02.728061 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 724/1462] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 71 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:02.729064 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e7225d-7045-4592-a97f-f122faf0e823 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:02.758819 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-38f301b4-119d-4506-8412-512109f1b4a5 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:02.760634 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-38f301b4-119d-4506-8412-512109f1b4a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:02.761817 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e7225d-7045-4592-a97f-f122faf0e823 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:02.767875 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39d77ee5-ec36-415d-9aec-000f9249e060 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:02.768728 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d77ee5-ec36-415d-9aec-000f9249e060 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:02.769091 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d77ee5-ec36-415d-9aec-000f9249e060 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:02.769451 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d77ee5-ec36-415d-9aec-000f9249e060 admin admin] PATH_INFO: `/v3/domains` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:02.771522 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39d77ee5-ec36-415d-9aec-000f9249e060 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 09:01:02.771845 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39d77ee5-ec36-415d-9aec-000f9249e060 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:02.773949 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39d77ee5-ec36-415d-9aec-000f9249e060 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:02.792213 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 739/1463] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:02.797108 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 725/1464] 10.222.0.36 () {58 vars in 1153 bytes} [Thu Oct 20 09:01:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 68 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 09:01:02.799287 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae0c8f8c-5b3a-4791-bd6b-c379ba455f24 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:02.799503 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38f301b4-119d-4506-8412-512109f1b4a5 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:02.799627 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae0c8f8c-5b3a-4791-bd6b-c379ba455f24 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:02.799826 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae0c8f8c-5b3a-4791-bd6b-c379ba455f24 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:02.802141 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 726/1465] 10.222.0.36 () {54 vars in 921 bytes} [Thu Oct 20 09:01:02 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 09:01:02.803003 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae0c8f8c-5b3a-4791-bd6b-c379ba455f24 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:02.816515 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38f301b4-119d-4506-8412-512109f1b4a5 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:02.834758 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae0c8f8c-5b3a-4791-bd6b-c379ba455f24 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:02.842574 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-703eff28-36c6-4b9c-b16f-060e6a1783bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:02.843217 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703eff28-36c6-4b9c-b16f-060e6a1783bd admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:02.843780 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703eff28-36c6-4b9c-b16f-060e6a1783bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:02.843942 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703eff28-36c6-4b9c-b16f-060e6a1783bd admin admin] PATH_INFO: `/v3/domains` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:02.844668 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-703eff28-36c6-4b9c-b16f-060e6a1783bd admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 09:01:02.844988 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-703eff28-36c6-4b9c-b16f-060e6a1783bd admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:02.846891 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-703eff28-36c6-4b9c-b16f-060e6a1783bd admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:02.848594 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68156857-d070-44b0-b5ba-b18d4be5d698 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:02.849391 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68156857-d070-44b0-b5ba-b18d4be5d698 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:02.849793 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68156857-d070-44b0-b5ba-b18d4be5d698 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:02.849938 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68156857-d070-44b0-b5ba-b18d4be5d698 admin admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:02.850929 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68156857-d070-44b0-b5ba-b18d4be5d698 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 09:01:02.851163 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68156857-d070-44b0-b5ba-b18d4be5d698 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:02.853461 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68156857-d070-44b0-b5ba-b18d4be5d698 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:02.864679 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 727/1466] 10.222.0.36 () {58 vars in 1153 bytes} [Thu Oct 20 09:01:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 62 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 09:01:02.865546 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-703eff28-36c6-4b9c-b16f-060e6a1783bd admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:02.886276 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-703eff28-36c6-4b9c-b16f-060e6a1783bd admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:02.892802 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ddf3600-b53e-411c-b2d9-99732c7e8425 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:02.893183 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 740/1467] 10.222.0.36 () {58 vars in 1153 bytes} [Thu Oct 20 09:01:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 101 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 09:01:02.893955 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ddf3600-b53e-411c-b2d9-99732c7e8425 admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:02.894303 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ddf3600-b53e-411c-b2d9-99732c7e8425 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:02.894683 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ddf3600-b53e-411c-b2d9-99732c7e8425 admin admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:02.895390 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ddf3600-b53e-411c-b2d9-99732c7e8425 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:02.897496 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ddf3600-b53e-411c-b2d9-99732c7e8425 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:02.897627 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68156857-d070-44b0-b5ba-b18d4be5d698 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:02.936971 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68156857-d070-44b0-b5ba-b18d4be5d698 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:02.947940 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc15aee7-8ea4-4d34-990f-d8862e803320 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:02.948666 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc15aee7-8ea4-4d34-990f-d8862e803320 admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:02.949302 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 728/1468] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:01:02 2022] POST /identity/v3/projects => generated 385 bytes in 84 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:01:02.949786 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc15aee7-8ea4-4d34-990f-d8862e803320 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:02.950580 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ddf3600-b53e-411c-b2d9-99732c7e8425 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:02.951029 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc15aee7-8ea4-4d34-990f-d8862e803320 admin admin] PATH_INFO: `/v3/projects` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:02.952096 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc15aee7-8ea4-4d34-990f-d8862e803320 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:02.956273 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc15aee7-8ea4-4d34-990f-d8862e803320 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:02.981997 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=89,keystone:INSERT=3,keystone:DELETE=8 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:03.016683 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 741/1469] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:01:02 2022] POST /identity/v3/projects => generated 379 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:01:03.018880 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc15aee7-8ea4-4d34-990f-d8862e803320 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:03.070037 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:03.104380 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc15aee7-8ea4-4d34-990f-d8862e803320 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:03.109672 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ec4aa63-7092-46c7-a24b-5c5e1d852594 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:03.110355 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec4aa63-7092-46c7-a24b-5c5e1d852594 admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:03.111011 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec4aa63-7092-46c7-a24b-5c5e1d852594 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:03.111482 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec4aa63-7092-46c7-a24b-5c5e1d852594 admin admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:03.112235 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ec4aa63-7092-46c7-a24b-5c5e1d852594 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:03.114606 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ec4aa63-7092-46c7-a24b-5c5e1d852594 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:03.143804 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ddf3600-b53e-411c-b2d9-99732c7e8425 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:03.157417 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 742/1470] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:01:03 2022] POST /identity/v3/users => generated 293 bytes in 137 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:01:03.157417 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ec4aa63-7092-46c7-a24b-5c5e1d852594 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:03.161771 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-621c48bf-98f7-471e-bbe5-8b7e7c566e9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:03.163742 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621c48bf-98f7-471e-bbe5-8b7e7c566e9e admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:03.165288 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621c48bf-98f7-471e-bbe5-8b7e7c566e9e admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:03.165737 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621c48bf-98f7-471e-bbe5-8b7e7c566e9e admin admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:03.166568 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-621c48bf-98f7-471e-bbe5-8b7e7c566e9e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:03.172443 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-621c48bf-98f7-471e-bbe5-8b7e7c566e9e admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:03.175195 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ec4aa63-7092-46c7-a24b-5c5e1d852594 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:03.189842 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc248822-9922-4cbe-9999-0eed2aedb98b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:03.190964 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc248822-9922-4cbe-9999-0eed2aedb98b admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:03.191399 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc248822-9922-4cbe-9999-0eed2aedb98b admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:03.191826 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc248822-9922-4cbe-9999-0eed2aedb98b admin admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:03.193225 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc248822-9922-4cbe-9999-0eed2aedb98b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:03.195599 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc248822-9922-4cbe-9999-0eed2aedb98b admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:03.233924 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 729/1471] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:01:02 2022] POST /identity/v3/projects => generated 385 bytes in 281 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:01:03.233924 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05fcaf4f-13bb-478a-9c79-57e936390f56 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:03.233924 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05fcaf4f-13bb-478a-9c79-57e936390f56 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:03.234298 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05fcaf4f-13bb-478a-9c79-57e936390f56 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:03.249855 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 743/1472] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:01:03 2022] POST /identity/v3/users => generated 290 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:01:03.250940 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc248822-9922-4cbe-9999-0eed2aedb98b admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:03.264951 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc248822-9922-4cbe-9999-0eed2aedb98b admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:03.272390 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-05fcaf4f-13bb-478a-9c79-57e936390f56 None None] MFA Rules not processed for user `a94e868115ec49918eda10724263930c`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:03.274169 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-05fcaf4f-13bb-478a-9c79-57e936390f56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:03.446569 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eabf734-c8b5-4f92-8e28-2ae713e38a0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:03.447437 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eabf734-c8b5-4f92-8e28-2ae713e38a0a admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:03.448682 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eabf734-c8b5-4f92-8e28-2ae713e38a0a admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:03.449124 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eabf734-c8b5-4f92-8e28-2ae713e38a0a admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:03.449953 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eabf734-c8b5-4f92-8e28-2ae713e38a0a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:03.452365 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eabf734-c8b5-4f92-8e28-2ae713e38a0a admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:03.477950 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 744/1473] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:01:03 2022] GET /identity/v3/roles => generated 1420 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:01:03.479371 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eabf734-c8b5-4f92-8e28-2ae713e38a0a admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:03.486843 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 730/1474] 10.222.0.36 () {58 vars in 1018 bytes} [Thu Oct 20 09:01:03 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 254 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:03.487356 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05fcaf4f-13bb-478a-9c79-57e936390f56 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:03.500483 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eabf734-c8b5-4f92-8e28-2ae713e38a0a admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:03.508402 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c05f3bd-699a-450a-9244-f12c00f97638 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:03.510527 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c05f3bd-699a-450a-9244-f12c00f97638 admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:03.511011 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c05f3bd-699a-450a-9244-f12c00f97638 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:03.511485 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c05f3bd-699a-450a-9244-f12c00f97638 admin admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:03.512322 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c05f3bd-699a-450a-9244-f12c00f97638 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:03.514511 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c05f3bd-699a-450a-9244-f12c00f97638 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:03.514643 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05fcaf4f-13bb-478a-9c79-57e936390f56 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:03.527748 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad4c5f81-2f73-476c-8537-ea462066a483 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:03.528583 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad4c5f81-2f73-476c-8537-ea462066a483 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:03.529552 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad4c5f81-2f73-476c-8537-ea462066a483 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:03.529843 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad4c5f81-2f73-476c-8537-ea462066a483 admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:03.531190 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad4c5f81-2f73-476c-8537-ea462066a483 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:03.534500 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad4c5f81-2f73-476c-8537-ea462066a483 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:03.550946 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 731/1475] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:01:03 2022] GET /identity/v3/roles => generated 1420 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:01:03.553211 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad4c5f81-2f73-476c-8537-ea462066a483 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:03.577254 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 745/1476] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:01:03 2022] POST /identity/v3/users => generated 293 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:01:03.578620 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c05f3bd-699a-450a-9244-f12c00f97638 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:03.579587 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad4c5f81-2f73-476c-8537-ea462066a483 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:03.592610 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5dbe260-df3d-43de-bb3c-b615b7ef27c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:03.593242 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5dbe260-df3d-43de-bb3c-b615b7ef27c8 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:03.593525 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5dbe260-df3d-43de-bb3c-b615b7ef27c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:03.593771 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5dbe260-df3d-43de-bb3c-b615b7ef27c8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:03.594773 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5dbe260-df3d-43de-bb3c-b615b7ef27c8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:03.597534 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5dbe260-df3d-43de-bb3c-b615b7ef27c8 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:03.609413 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c05f3bd-699a-450a-9244-f12c00f97638 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:03.620966 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 732/1477] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:01:03 2022] GET /identity/v3/roles => generated 1420 bytes in 70 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:01:03.622010 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5dbe260-df3d-43de-bb3c-b615b7ef27c8 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:03.629611 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-047d5d9b-da66-4fd6-bc58-f2d5ef74c5c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:03.631854 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-047d5d9b-da66-4fd6-bc58-f2d5ef74c5c7 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:03.632358 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-047d5d9b-da66-4fd6-bc58-f2d5ef74c5c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:03.634726 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-047d5d9b-da66-4fd6-bc58-f2d5ef74c5c7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:03.635077 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5dbe260-df3d-43de-bb3c-b615b7ef27c8 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:03.636722 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-047d5d9b-da66-4fd6-bc58-f2d5ef74c5c7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:03.641105 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-047d5d9b-da66-4fd6-bc58-f2d5ef74c5c7 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:03.641331 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae72f400-650b-47d1-b508-c21ae8f94271 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:03.641786 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae72f400-650b-47d1-b508-c21ae8f94271 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:03.642065 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae72f400-650b-47d1-b508-c21ae8f94271 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:03.642342 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae72f400-650b-47d1-b508-c21ae8f94271 admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:03.643027 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae72f400-650b-47d1-b508-c21ae8f94271 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:03.644818 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae72f400-650b-47d1-b508-c21ae8f94271 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:03.667864 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 733/1478] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:01:03 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:01:03.671188 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae72f400-650b-47d1-b508-c21ae8f94271 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:03.682855 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 746/1479] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:01:03 2022] GET /identity/v3/roles => generated 1420 bytes in 105 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:01:03.683745 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-047d5d9b-da66-4fd6-bc58-f2d5ef74c5c7 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:03.698823 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae72f400-650b-47d1-b508-c21ae8f94271 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:03.704792 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-047d5d9b-da66-4fd6-bc58-f2d5ef74c5c7 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:03.713830 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd03624c-82f5-423a-93f0-ee755afa07d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:03.715761 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd03624c-82f5-423a-93f0-ee755afa07d9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:03.716067 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd03624c-82f5-423a-93f0-ee755afa07d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:03.716304 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd03624c-82f5-423a-93f0-ee755afa07d9 admin admin] PATH_INFO: `/v3/projects/418f119557ff4d919ebb4a4f9e7be922/users/4765f2cfd7bc4b01bec5ac30580fa102/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:03.746823 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be0c7681-1309-4079-a80b-f5cc36bb3406 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:03.748245 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be0c7681-1309-4079-a80b-f5cc36bb3406 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:03.748651 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be0c7681-1309-4079-a80b-f5cc36bb3406 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:03.749122 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be0c7681-1309-4079-a80b-f5cc36bb3406 admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:03.749293 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd03624c-82f5-423a-93f0-ee755afa07d9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=418f119557ff4d919ebb4a4f9e7be922, user_id=4765f2cfd7bc4b01bec5ac30580fa102, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:03.752935 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd03624c-82f5-423a-93f0-ee755afa07d9 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:03.753663 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be0c7681-1309-4079-a80b-f5cc36bb3406 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:03.756611 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be0c7681-1309-4079-a80b-f5cc36bb3406 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:03.786712 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 734/1480] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:01:03 2022] PUT /identity/v3/projects/418f119557ff4d919ebb4a4f9e7be922/users/4765f2cfd7bc4b01bec5ac30580fa102/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:03.790180 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd03624c-82f5-423a-93f0-ee755afa07d9 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:03.804630 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 747/1481] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:01:03 2022] GET /identity/v3/roles => generated 1420 bytes in 121 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:01:03.806882 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e734ad-979d-4ba9-aa59-f7cdff13f299 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:03.813776 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e734ad-979d-4ba9-aa59-f7cdff13f299 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:03.818429 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e734ad-979d-4ba9-aa59-f7cdff13f299 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:03.857403 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:03.929359 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd03624c-82f5-423a-93f0-ee755afa07d9 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:03.936773 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-840b4ff7-851b-4561-b4e3-0f99daf8a104 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:03.937979 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840b4ff7-851b-4561-b4e3-0f99daf8a104 admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:03.938434 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840b4ff7-851b-4561-b4e3-0f99daf8a104 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:03.938990 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840b4ff7-851b-4561-b4e3-0f99daf8a104 admin admin] PATH_INFO: `/v3/projects/7a8bb34c215647bc98941e0acd5a6da7/users/035b8a17c4594f279853c63ed2cd00c5/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:03.948072 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-90e734ad-979d-4ba9-aa59-f7cdff13f299 None None] MFA Rules not processed for user `4765f2cfd7bc4b01bec5ac30580fa102`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:03.952037 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-90e734ad-979d-4ba9-aa59-f7cdff13f299 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:03.977226 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-840b4ff7-851b-4561-b4e3-0f99daf8a104 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7a8bb34c215647bc98941e0acd5a6da7, user_id=035b8a17c4594f279853c63ed2cd00c5, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:03.982489 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-840b4ff7-851b-4561-b4e3-0f99daf8a104 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:04.023225 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 735/1482] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:01:03 2022] PUT /identity/v3/projects/7a8bb34c215647bc98941e0acd5a6da7/users/035b8a17c4594f279853c63ed2cd00c5/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 236 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:04.024222 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-840b4ff7-851b-4561-b4e3-0f99daf8a104 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:04.133966 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=91,keystone:INSERT=9 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:04.199266 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 748/1483] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:03 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 394 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:04.200236 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90e734ad-979d-4ba9-aa59-f7cdff13f299 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:04.205643 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-90e734ad-979d-4ba9-aa59-f7cdff13f299 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:04.286383 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-840b4ff7-851b-4561-b4e3-0f99daf8a104 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:04.293409 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5813251-0d63-4aa1-ac6b-1ce73d7c9714 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:04.314169 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5813251-0d63-4aa1-ac6b-1ce73d7c9714 admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:04.315112 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5813251-0d63-4aa1-ac6b-1ce73d7c9714 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:04.316336 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5813251-0d63-4aa1-ac6b-1ce73d7c9714 admin admin] PATH_INFO: `/v3/projects/fc728a8b82294937b1dcc1edb6528e36/users/bb80cbbd4c684beca47007ba2f9f559e/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:04.353600 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5813251-0d63-4aa1-ac6b-1ce73d7c9714 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fc728a8b82294937b1dcc1edb6528e36, user_id=bb80cbbd4c684beca47007ba2f9f559e, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:04.357426 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5813251-0d63-4aa1-ac6b-1ce73d7c9714 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:04.410369 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 736/1484] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:01:04 2022] PUT /identity/v3/projects/fc728a8b82294937b1dcc1edb6528e36/users/bb80cbbd4c684beca47007ba2f9f559e/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 386 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:04.417578 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e46e096-44ef-41f0-824f-72d1f810cd5c None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:04.419116 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e46e096-44ef-41f0-824f-72d1f810cd5c None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:04.420644 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e46e096-44ef-41f0-824f-72d1f810cd5c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:04.422969 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90e734ad-979d-4ba9-aa59-f7cdff13f299 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:04.536477 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-8e46e096-44ef-41f0-824f-72d1f810cd5c None None] MFA Rules not processed for user `035b8a17c4594f279853c63ed2cd00c5`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:04.548692 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-8e46e096-44ef-41f0-824f-72d1f810cd5c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:04.550041 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43cb4bc2-f5c6-49fc-84ca-4867c3e6a482 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a94e868115ec49918eda10724263930c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:04.551743 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43cb4bc2-f5c6-49fc-84ca-4867c3e6a482 service nova] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:04.552252 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43cb4bc2-f5c6-49fc-84ca-4867c3e6a482 service nova] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:04.552762 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43cb4bc2-f5c6-49fc-84ca-4867c3e6a482 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:04.567571 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43cb4bc2-f5c6-49fc-84ca-4867c3e6a482 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:04.569715 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43cb4bc2-f5c6-49fc-84ca-4867c3e6a482 service nova] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:04.599985 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 749/1485] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:01:04 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 400 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:01:04.607736 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43cb4bc2-f5c6-49fc-84ca-4867c3e6a482 service nova] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:04.612114 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-43cb4bc2-f5c6-49fc-84ca-4867c3e6a482 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:04.787971 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43cb4bc2-f5c6-49fc-84ca-4867c3e6a482 service nova] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:04.796377 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3721063f-00b1-4470-82cc-a2813aad8ffc service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:04.803142 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3721063f-00b1-4470-82cc-a2813aad8ffc service cinder] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:04.803417 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3721063f-00b1-4470-82cc-a2813aad8ffc service cinder] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:04.803700 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3721063f-00b1-4470-82cc-a2813aad8ffc service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:04.849878 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 737/1486] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:04 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 439 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:04.851783 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a032e3-98c4-4c56-aba7-194037cca5ef None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:04.852956 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a032e3-98c4-4c56-aba7-194037cca5ef None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:04.854673 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a032e3-98c4-4c56-aba7-194037cca5ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:04.870904 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:04.921266 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-17a032e3-98c4-4c56-aba7-194037cca5ef None None] MFA Rules not processed for user `bb80cbbd4c684beca47007ba2f9f559e`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:04.922506 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-17a032e3-98c4-4c56-aba7-194037cca5ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:04.933593 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3721063f-00b1-4470-82cc-a2813aad8ffc service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:04.940250 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3721063f-00b1-4470-82cc-a2813aad8ffc service cinder] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:04.953230 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:04.979480 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 750/1487] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:01:04 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 379 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:01:04.981323 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3721063f-00b1-4470-82cc-a2813aad8ffc service cinder] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:05.126603 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 738/1488] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:04 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 276 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:05.128110 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17a032e3-98c4-4c56-aba7-194037cca5ef None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:05.133553 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3721063f-00b1-4470-82cc-a2813aad8ffc service cinder] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:05.142638 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6b9d37f-7b01-4928-9623-8e7c0c385412 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:05.145996 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b9d37f-7b01-4928-9623-8e7c0c385412 admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:05.145996 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b9d37f-7b01-4928-9623-8e7c0c385412 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:05.145996 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b9d37f-7b01-4928-9623-8e7c0c385412 admin admin] PATH_INFO: `/v3/projects` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:05.146364 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6b9d37f-7b01-4928-9623-8e7c0c385412 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:05.151786 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6b9d37f-7b01-4928-9623-8e7c0c385412 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:05.157753 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17a032e3-98c4-4c56-aba7-194037cca5ef None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:05.163333 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1193083d-24a0-48fa-89e3-b2a8838afd13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:05.164156 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1193083d-24a0-48fa-89e3-b2a8838afd13 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:05.164363 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1193083d-24a0-48fa-89e3-b2a8838afd13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:05.164647 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1193083d-24a0-48fa-89e3-b2a8838afd13 admin admin] PATH_INFO: `/v3/users/4765f2cfd7bc4b01bec5ac30580fa102` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:05.175375 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1193083d-24a0-48fa-89e3-b2a8838afd13 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4765f2cfd7bc4b01bec5ac30580fa102)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:05.177362 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1193083d-24a0-48fa-89e3-b2a8838afd13 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:05.184859 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 751/1489] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:01:04 2022] POST /identity/v3/projects => generated 381 bytes in 205 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:01:05.185925 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6b9d37f-7b01-4928-9623-8e7c0c385412 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:05.262274 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-1193083d-24a0-48fa-89e3-b2a8838afd13 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4765f2cfd7bc4b01bec5ac30580fa102'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:05.279830 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-1193083d-24a0-48fa-89e3-b2a8838afd13 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4765f2cfd7bc4b01bec5ac30580fa102'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:05.297231 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-1193083d-24a0-48fa-89e3-b2a8838afd13 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4765f2cfd7bc4b01bec5ac30580fa102'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:05.301595 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6b9d37f-7b01-4928-9623-8e7c0c385412 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:05.306813 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-1193083d-24a0-48fa-89e3-b2a8838afd13 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4765f2cfd7bc4b01bec5ac30580fa102'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:05.307561 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-1193083d-24a0-48fa-89e3-b2a8838afd13 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4765f2cfd7bc4b01bec5ac30580fa102'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:05.308367 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-841030b3-066e-45d7-b17f-95f6254bc906 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:05.309231 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841030b3-066e-45d7-b17f-95f6254bc906 admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:05.309754 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841030b3-066e-45d7-b17f-95f6254bc906 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:05.310365 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841030b3-066e-45d7-b17f-95f6254bc906 admin admin] PATH_INFO: `/v3/projects` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:05.311245 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-841030b3-066e-45d7-b17f-95f6254bc906 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:05.318621 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 739/1490] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:01:05 2022] DELETE /identity/v3/users/4765f2cfd7bc4b01bec5ac30580fa102 => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:05.318621 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1193083d-24a0-48fa-89e3-b2a8838afd13 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:05.318621 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-1193083d-24a0-48fa-89e3-b2a8838afd13 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:05.328719 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-841030b3-066e-45d7-b17f-95f6254bc906 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:05.414519 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 752/1491] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:01:05 2022] POST /identity/v3/projects => generated 385 bytes in 229 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:01:05.416127 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d72981-2a9c-42e8-ac27-cacf80dc884d None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:05.416412 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d72981-2a9c-42e8-ac27-cacf80dc884d None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:05.416681 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d72981-2a9c-42e8-ac27-cacf80dc884d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:05.434787 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1193083d-24a0-48fa-89e3-b2a8838afd13 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:05.446292 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-47d72981-2a9c-42e8-ac27-cacf80dc884d None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:05.447802 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-47d72981-2a9c-42e8-ac27-cacf80dc884d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:05.503991 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:05.515025 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:05.534171 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 753/1492] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:05.535232 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d72981-2a9c-42e8-ac27-cacf80dc884d None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:05.539394 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-47d72981-2a9c-42e8-ac27-cacf80dc884d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:05.557324 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d72981-2a9c-42e8-ac27-cacf80dc884d None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:05.563575 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f50a8df-7fbb-43f6-ac1c-ab2853224592 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:05.564638 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f50a8df-7fbb-43f6-ac1c-ab2853224592 admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:05.565098 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f50a8df-7fbb-43f6-ac1c-ab2853224592 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:05.565556 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f50a8df-7fbb-43f6-ac1c-ab2853224592 admin admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:05.566529 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f50a8df-7fbb-43f6-ac1c-ab2853224592 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:05.568733 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f50a8df-7fbb-43f6-ac1c-ab2853224592 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:05.579757 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e38081cd-309d-439c-8656-4b8ae29b700c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:05.580384 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e38081cd-309d-439c-8656-4b8ae29b700c admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:05.580667 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e38081cd-309d-439c-8656-4b8ae29b700c admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:05.580943 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e38081cd-309d-439c-8656-4b8ae29b700c admin admin] PATH_INFO: `/v3/users` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:05.582436 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e38081cd-309d-439c-8656-4b8ae29b700c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:05.585268 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e38081cd-309d-439c-8656-4b8ae29b700c admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:05.629665 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 740/1493] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:01:05 2022] POST /identity/v3/users => generated 297 bytes in 313 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:01:05.630687 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e38081cd-309d-439c-8656-4b8ae29b700c admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:05.635859 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-e38081cd-309d-439c-8656-4b8ae29b700c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:05.639446 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 754/1494] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:01:05 2022] POST /identity/v3/users => generated 299 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:01:05.640494 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f50a8df-7fbb-43f6-ac1c-ab2853224592 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:05.664675 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f50a8df-7fbb-43f6-ac1c-ab2853224592 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:05.681419 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c951e7fc-0d42-4c18-8fcb-9a6c25dc362b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:05.683780 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c951e7fc-0d42-4c18-8fcb-9a6c25dc362b admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:05.684256 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c951e7fc-0d42-4c18-8fcb-9a6c25dc362b admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:05.684611 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c951e7fc-0d42-4c18-8fcb-9a6c25dc362b admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:05.685753 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c951e7fc-0d42-4c18-8fcb-9a6c25dc362b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:05.701299 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c951e7fc-0d42-4c18-8fcb-9a6c25dc362b admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:05.728507 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 755/1495] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:01:05 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:01:05.730752 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c951e7fc-0d42-4c18-8fcb-9a6c25dc362b admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:05.731168 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e38081cd-309d-439c-8656-4b8ae29b700c admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:05.735850 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-320653be-ea38-46a4-afe0-cf891ca4e05e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:05.736465 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320653be-ea38-46a4-afe0-cf891ca4e05e service neutron] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:05.736785 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320653be-ea38-46a4-afe0-cf891ca4e05e service neutron] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:05.737105 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320653be-ea38-46a4-afe0-cf891ca4e05e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:05.748052 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c951e7fc-0d42-4c18-8fcb-9a6c25dc362b admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:05.753540 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a7cba2d-3734-46a5-95cf-1af894601b39 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:05.754117 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a7cba2d-3734-46a5-95cf-1af894601b39 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:05.754398 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a7cba2d-3734-46a5-95cf-1af894601b39 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:05.754715 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a7cba2d-3734-46a5-95cf-1af894601b39 admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:05.756182 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a7cba2d-3734-46a5-95cf-1af894601b39 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:05.757716 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a7cba2d-3734-46a5-95cf-1af894601b39 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:05.759384 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-320653be-ea38-46a4-afe0-cf891ca4e05e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:05.761208 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-320653be-ea38-46a4-afe0-cf891ca4e05e service neutron] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:05.787876 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 756/1496] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:01:05 2022] GET /identity/v3/roles => generated 1420 bytes in 59 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:01:05.789660 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a7cba2d-3734-46a5-95cf-1af894601b39 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:05.800183 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 741/1497] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:01:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:01:05.802047 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-320653be-ea38-46a4-afe0-cf891ca4e05e service neutron] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:05.811140 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a7cba2d-3734-46a5-95cf-1af894601b39 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:05.821655 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0f5ecf8-dfa5-4432-85f5-b8ecf1ac9bdb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:05.822610 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f5ecf8-dfa5-4432-85f5-b8ecf1ac9bdb admin admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:05.824174 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f5ecf8-dfa5-4432-85f5-b8ecf1ac9bdb admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:05.824693 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f5ecf8-dfa5-4432-85f5-b8ecf1ac9bdb admin admin] PATH_INFO: `/v3/projects/624b56c30c8e47d392e621a569ea0bbb/users/ee807b6eec054431a863a972ce8ad67b/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:05.827710 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-320653be-ea38-46a4-afe0-cf891ca4e05e service neutron] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:05.836333 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d0879ea-a4e3-40e1-a28e-5ed7db6f49c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:05.837573 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d0879ea-a4e3-40e1-a28e-5ed7db6f49c7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:05.838019 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d0879ea-a4e3-40e1-a28e-5ed7db6f49c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:05.838580 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d0879ea-a4e3-40e1-a28e-5ed7db6f49c7 admin admin] PATH_INFO: `/v3/projects/ce934e25f1ed422abf53d5e384c01518/users/978ebf3bc10d45d5bb251ebd10b28a29/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:05.851317 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0f5ecf8-dfa5-4432-85f5-b8ecf1ac9bdb admin admin] RBAC: Authorizing `identity:create_grant(project_id=624b56c30c8e47d392e621a569ea0bbb, user_id=ee807b6eec054431a863a972ce8ad67b, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:05.860675 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0f5ecf8-dfa5-4432-85f5-b8ecf1ac9bdb admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:05.864821 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d0879ea-a4e3-40e1-a28e-5ed7db6f49c7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ce934e25f1ed422abf53d5e384c01518, user_id=978ebf3bc10d45d5bb251ebd10b28a29, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:05.869434 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d0879ea-a4e3-40e1-a28e-5ed7db6f49c7 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:05.890159 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 742/1498] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:01:05 2022] PUT /identity/v3/projects/ce934e25f1ed422abf53d5e384c01518/users/978ebf3bc10d45d5bb251ebd10b28a29/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 89 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:05.891461 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 757/1499] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:01:05 2022] PUT /identity/v3/projects/624b56c30c8e47d392e621a569ea0bbb/users/ee807b6eec054431a863a972ce8ad67b/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:05.894070 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d0879ea-a4e3-40e1-a28e-5ed7db6f49c7 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:05.895285 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0f5ecf8-dfa5-4432-85f5-b8ecf1ac9bdb admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:06.008579 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0f5ecf8-dfa5-4432-85f5-b8ecf1ac9bdb admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:06.065340 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d0879ea-a4e3-40e1-a28e-5ed7db6f49c7 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:06.076180 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-846d36ea-8364-450c-bde4-56596cfa2385 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:06.076872 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-846d36ea-8364-450c-bde4-56596cfa2385 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:06.077150 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-846d36ea-8364-450c-bde4-56596cfa2385 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:06.078052 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-846d36ea-8364-450c-bde4-56596cfa2385 admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:06.078052 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-846d36ea-8364-450c-bde4-56596cfa2385 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:06.081493 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-846d36ea-8364-450c-bde4-56596cfa2385 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:06.107891 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 743/1500] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:01:05 2022] GET /identity/v3/roles => generated 1420 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:01:06.111256 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20261042-75c9-4ddd-a272-fcde4d1c1575 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:06.111545 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20261042-75c9-4ddd-a272-fcde4d1c1575 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:06.111809 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20261042-75c9-4ddd-a272-fcde4d1c1575 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:06.114498 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 744/1501] 10.222.0.36 () {54 vars in 921 bytes} [Thu Oct 20 09:01:06 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 09:01:06.115242 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20261042-75c9-4ddd-a272-fcde4d1c1575 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:06.146860 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f381663f-d2b7-4dc9-9d57-bcd56c74380a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:06.148130 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20261042-75c9-4ddd-a272-fcde4d1c1575 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:06.149608 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f381663f-d2b7-4dc9-9d57-bcd56c74380a admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:06.150241 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f381663f-d2b7-4dc9-9d57-bcd56c74380a admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:06.150589 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f381663f-d2b7-4dc9-9d57-bcd56c74380a admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:06.152028 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f381663f-d2b7-4dc9-9d57-bcd56c74380a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:06.154046 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f381663f-d2b7-4dc9-9d57-bcd56c74380a admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:06.166019 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1fdc615-b8b9-4311-8b8e-82ddcfb214fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:06.166019 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1fdc615-b8b9-4311-8b8e-82ddcfb214fb admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:06.166019 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1fdc615-b8b9-4311-8b8e-82ddcfb214fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:06.166019 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1fdc615-b8b9-4311-8b8e-82ddcfb214fb admin admin] PATH_INFO: `/v3/projects/ce934e25f1ed422abf53d5e384c01518/users/978ebf3bc10d45d5bb251ebd10b28a29/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:06.168801 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1fdc615-b8b9-4311-8b8e-82ddcfb214fb admin admin] RBAC: Authorizing `identity:create_grant(project_id=ce934e25f1ed422abf53d5e384c01518, user_id=978ebf3bc10d45d5bb251ebd10b28a29, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:06.173200 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1fdc615-b8b9-4311-8b8e-82ddcfb214fb admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:06.184610 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 758/1502] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:01:05 2022] GET /identity/v3/roles => generated 1420 bytes in 288 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 20 09:01:06.186307 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f381663f-d2b7-4dc9-9d57-bcd56c74380a admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:06.201179 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 745/1503] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:01:06 2022] PUT /identity/v3/projects/ce934e25f1ed422abf53d5e384c01518/users/978ebf3bc10d45d5bb251ebd10b28a29/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 86 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:06.204815 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f381663f-d2b7-4dc9-9d57-bcd56c74380a admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:06.213825 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f006fb3d-b1a5-40f7-b9e0-725468358d33 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:06.214111 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f006fb3d-b1a5-40f7-b9e0-725468358d33 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:06.214385 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f006fb3d-b1a5-40f7-b9e0-725468358d33 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:06.266633 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:06.284971 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-f006fb3d-b1a5-40f7-b9e0-725468358d33 None None] MFA Rules not processed for user `978ebf3bc10d45d5bb251ebd10b28a29`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:06.286238 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-f006fb3d-b1a5-40f7-b9e0-725468358d33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:06.317277 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:06.362784 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b9cadd1-21c7-4103-b5fb-06f5551de3f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:06.370077 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b9cadd1-21c7-4103-b5fb-06f5551de3f9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:06.370709 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b9cadd1-21c7-4103-b5fb-06f5551de3f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:06.370709 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b9cadd1-21c7-4103-b5fb-06f5551de3f9 admin admin] PATH_INFO: `/v3/projects/624b56c30c8e47d392e621a569ea0bbb/users/ee807b6eec054431a863a972ce8ad67b/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:06.382515 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b9cadd1-21c7-4103-b5fb-06f5551de3f9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=624b56c30c8e47d392e621a569ea0bbb, user_id=ee807b6eec054431a863a972ce8ad67b, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:06.385424 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b9cadd1-21c7-4103-b5fb-06f5551de3f9 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:06.413263 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 759/1504] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:01:06 2022] PUT /identity/v3/projects/624b56c30c8e47d392e621a569ea0bbb/users/ee807b6eec054431a863a972ce8ad67b/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 227 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:06.415180 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb2647c-bbce-4e3d-9b04-20a34f15f61b None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:06.415467 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb2647c-bbce-4e3d-9b04-20a34f15f61b None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:06.415741 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb2647c-bbce-4e3d-9b04-20a34f15f61b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:06.464495 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 746/1505] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:06 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 252 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:06.465228 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-5eb2647c-bbce-4e3d-9b04-20a34f15f61b None None] MFA Rules not processed for user `a94e868115ec49918eda10724263930c`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:06.466501 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-5eb2647c-bbce-4e3d-9b04-20a34f15f61b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:06.467372 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d942769a-d40f-459a-ada5-b8db3822cafb None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:06.467957 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d942769a-d40f-459a-ada5-b8db3822cafb None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:06.474897 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d942769a-d40f-459a-ada5-b8db3822cafb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:06.518435 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-d942769a-d40f-459a-ada5-b8db3822cafb None None] MFA Rules not processed for user `ee807b6eec054431a863a972ce8ad67b`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:06.519987 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-d942769a-d40f-459a-ada5-b8db3822cafb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:06.568537 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 760/1506] 10.222.0.36 () {58 vars in 1018 bytes} [Thu Oct 20 09:01:06 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:06.569798 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eb2647c-bbce-4e3d-9b04-20a34f15f61b None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:06.573188 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-5eb2647c-bbce-4e3d-9b04-20a34f15f61b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:06.627386 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 747/1507] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:06 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:06.652524 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d942769a-d40f-459a-ada5-b8db3822cafb None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:06.655609 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-d942769a-d40f-459a-ada5-b8db3822cafb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:06.660541 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eb2647c-bbce-4e3d-9b04-20a34f15f61b None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:06.665821 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcdd4ead-53cc-41b3-b04c-9838830fd66b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:06.666687 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcdd4ead-53cc-41b3-b04c-9838830fd66b service cinder] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:06.667958 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcdd4ead-53cc-41b3-b04c-9838830fd66b service cinder] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:06.667958 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcdd4ead-53cc-41b3-b04c-9838830fd66b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:06.671223 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d942769a-d40f-459a-ada5-b8db3822cafb None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:06.675899 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82561222-7f30-495e-ba37-8c83088f4b78 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:06.676627 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82561222-7f30-495e-ba37-8c83088f4b78 service cinder] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:06.677052 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82561222-7f30-495e-ba37-8c83088f4b78 service cinder] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:06.677550 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82561222-7f30-495e-ba37-8c83088f4b78 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:06.692587 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82561222-7f30-495e-ba37-8c83088f4b78 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:06.695209 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82561222-7f30-495e-ba37-8c83088f4b78 service cinder] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:06.727539 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 748/1508] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:01:06 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 75 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:01:06.734113 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82561222-7f30-495e-ba37-8c83088f4b78 service cinder] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:06.736749 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-82561222-7f30-495e-ba37-8c83088f4b78 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:06.765924 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82561222-7f30-495e-ba37-8c83088f4b78 service cinder] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:06.773591 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2605833-d913-49f3-af05-c6991fe6b2d1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a94e868115ec49918eda10724263930c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:06.774523 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2605833-d913-49f3-af05-c6991fe6b2d1 service nova] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:06.774951 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2605833-d913-49f3-af05-c6991fe6b2d1 service nova] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:06.775471 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2605833-d913-49f3-af05-c6991fe6b2d1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:06.796426 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2605833-d913-49f3-af05-c6991fe6b2d1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:06.797797 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcdd4ead-53cc-41b3-b04c-9838830fd66b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:06.798773 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2605833-d913-49f3-af05-c6991fe6b2d1 service nova] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:06.799674 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcdd4ead-53cc-41b3-b04c-9838830fd66b service cinder] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:06.804306 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:06.830125 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 749/1509] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:01:06 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 20 09:01:06.836908 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 761/1510] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:01:06 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 267 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:01:06.877661 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2605833-d913-49f3-af05-c6991fe6b2d1 service nova] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:06.900445 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcdd4ead-53cc-41b3-b04c-9838830fd66b service cinder] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:06.902164 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-bcdd4ead-53cc-41b3-b04c-9838830fd66b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:07.048002 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2605833-d913-49f3-af05-c6991fe6b2d1 service nova] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:07.060987 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3efede0-0358-444d-8a04-f05730481b8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:07.061645 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3efede0-0358-444d-8a04-f05730481b8e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:07.061918 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3efede0-0358-444d-8a04-f05730481b8e admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:07.062185 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3efede0-0358-444d-8a04-f05730481b8e admin admin] PATH_INFO: `/v3/users/bb80cbbd4c684beca47007ba2f9f559e` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:07.064394 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcdd4ead-53cc-41b3-b04c-9838830fd66b service cinder] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:07.065255 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3efede0-0358-444d-8a04-f05730481b8e admin admin] RBAC: Authorizing `identity:delete_user(user_id=bb80cbbd4c684beca47007ba2f9f559e)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:07.067425 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3efede0-0358-444d-8a04-f05730481b8e admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:07.071790 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0fc066b-e428-48c2-be28-ea3f6ea2fe46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:07.074412 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0fc066b-e428-48c2-be28-ea3f6ea2fe46 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:07.074931 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0fc066b-e428-48c2-be28-ea3f6ea2fe46 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:07.075398 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0fc066b-e428-48c2-be28-ea3f6ea2fe46 admin admin] PATH_INFO: `/v3/projects/418f119557ff4d919ebb4a4f9e7be922` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:07.077769 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0fc066b-e428-48c2-be28-ea3f6ea2fe46 admin admin] RBAC: Authorizing `identity:delete_project(project_id=418f119557ff4d919ebb4a4f9e7be922)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:07.080911 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0fc066b-e428-48c2-be28-ea3f6ea2fe46 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:07.100236 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:07.217516 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-f3efede0-0358-444d-8a04-f05730481b8e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bb80cbbd4c684beca47007ba2f9f559e'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:07.222301 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-a0fc066b-e428-48c2-be28-ea3f6ea2fe46 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '418f119557ff4d919ebb4a4f9e7be922', 'request_id': 'req-a0fc066b-e428-48c2-be28-ea3f6ea2fe46'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:07.229495 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-f3efede0-0358-444d-8a04-f05730481b8e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bb80cbbd4c684beca47007ba2f9f559e'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:07.230838 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-a0fc066b-e428-48c2-be28-ea3f6ea2fe46 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '418f119557ff4d919ebb4a4f9e7be922', 'request_id': 'req-a0fc066b-e428-48c2-be28-ea3f6ea2fe46'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:07.241026 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-a0fc066b-e428-48c2-be28-ea3f6ea2fe46 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '418f119557ff4d919ebb4a4f9e7be922', 'request_id': 'req-a0fc066b-e428-48c2-be28-ea3f6ea2fe46'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:07.250363 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-f3efede0-0358-444d-8a04-f05730481b8e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bb80cbbd4c684beca47007ba2f9f559e'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:07.260027 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-a0fc066b-e428-48c2-be28-ea3f6ea2fe46 admin admin] The token cache is being invalidate because project 418f119557ff4d919ebb4a4f9e7be922 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81562) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:01:07.260316 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-a0fc066b-e428-48c2-be28-ea3f6ea2fe46 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:07.261717 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-a0fc066b-e428-48c2-be28-ea3f6ea2fe46 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:07.266878 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 762/1511] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:01:06 2022] DELETE /identity/v3/projects/418f119557ff4d919ebb4a4f9e7be922 => generated 0 bytes in 364 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:07.267268 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-f3efede0-0358-444d-8a04-f05730481b8e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bb80cbbd4c684beca47007ba2f9f559e'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:07.268190 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-f3efede0-0358-444d-8a04-f05730481b8e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bb80cbbd4c684beca47007ba2f9f559e'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:07.268531 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc65209-b11b-4a74-bab8-d5266b8e907c None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:07.268898 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc65209-b11b-4a74-bab8-d5266b8e907c None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:07.269261 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc65209-b11b-4a74-bab8-d5266b8e907c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:07.280880 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 750/1512] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:01:06 2022] DELETE /identity/v3/users/bb80cbbd4c684beca47007ba2f9f559e => generated 0 bytes in 403 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:07.282971 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bd81cf1-aa3d-4167-8f11-f1192e79bad3 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:07.283237 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bd81cf1-aa3d-4167-8f11-f1192e79bad3 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:07.283629 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bd81cf1-aa3d-4167-8f11-f1192e79bad3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:07.299299 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-0bc65209-b11b-4a74-bab8-d5266b8e907c None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:07.300649 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-0bc65209-b11b-4a74-bab8-d5266b8e907c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:07.309298 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-0bd81cf1-aa3d-4167-8f11-f1192e79bad3 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:07.311338 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-0bd81cf1-aa3d-4167-8f11-f1192e79bad3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:07.374746 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 763/1513] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:07.374746 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bc65209-b11b-4a74-bab8-d5266b8e907c None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:07.374746 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-0bc65209-b11b-4a74-bab8-d5266b8e907c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:07.395110 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 751/1514] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:07.397048 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df64fcc-cbbf-412f-855a-d9b94b8aac30 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:07.398103 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df64fcc-cbbf-412f-855a-d9b94b8aac30 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:07.398409 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df64fcc-cbbf-412f-855a-d9b94b8aac30 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:07.419485 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-0df64fcc-cbbf-412f-855a-d9b94b8aac30 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:07.420391 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-0df64fcc-cbbf-412f-855a-d9b94b8aac30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:07.420761 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:07.431306 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 752/1515] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:07.432264 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0df64fcc-cbbf-412f-855a-d9b94b8aac30 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:07.433644 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-0df64fcc-cbbf-412f-855a-d9b94b8aac30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:07.463868 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bc65209-b11b-4a74-bab8-d5266b8e907c None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:07.469140 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e975752-8fa3-40bf-aa75-399a5ccbedaa tempest-VolumeQuotaClassesTest-1435933842 tempest-VolumeQuotaClassesTest-1435933842-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ee807b6eec054431a863a972ce8ad67b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '624b56c30c8e47d392e621a569ea0bbb', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:07.469742 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e975752-8fa3-40bf-aa75-399a5ccbedaa tempest-VolumeQuotaClassesTest-1435933842 tempest-VolumeQuotaClassesTest-1435933842-project-admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:07.470065 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e975752-8fa3-40bf-aa75-399a5ccbedaa tempest-VolumeQuotaClassesTest-1435933842 tempest-VolumeQuotaClassesTest-1435933842-project-admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:07.470287 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e975752-8fa3-40bf-aa75-399a5ccbedaa tempest-VolumeQuotaClassesTest-1435933842 tempest-VolumeQuotaClassesTest-1435933842-project-admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:07.471489 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e975752-8fa3-40bf-aa75-399a5ccbedaa tempest-VolumeQuotaClassesTest-1435933842 tempest-VolumeQuotaClassesTest-1435933842-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 09:01:07.471489 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e975752-8fa3-40bf-aa75-399a5ccbedaa tempest-VolumeQuotaClassesTest-1435933842 tempest-VolumeQuotaClassesTest-1435933842-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:07.473013 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e975752-8fa3-40bf-aa75-399a5ccbedaa tempest-VolumeQuotaClassesTest-1435933842 tempest-VolumeQuotaClassesTest-1435933842-project-admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:07.489216 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 764/1516] 10.222.0.36 () {58 vars in 1153 bytes} [Thu Oct 20 09:01:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 115 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 09:01:07.489216 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e975752-8fa3-40bf-aa75-399a5ccbedaa tempest-VolumeQuotaClassesTest-1435933842 tempest-VolumeQuotaClassesTest-1435933842-project-admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:07.499898 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0df64fcc-cbbf-412f-855a-d9b94b8aac30 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:07.500297 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e975752-8fa3-40bf-aa75-399a5ccbedaa tempest-VolumeQuotaClassesTest-1435933842 tempest-VolumeQuotaClassesTest-1435933842-project-admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:07.503479 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8157ba61-1f4d-429d-be8d-041654caceb9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:07.504087 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8157ba61-1f4d-429d-be8d-041654caceb9 service neutron] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:07.504361 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8157ba61-1f4d-429d-be8d-041654caceb9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:07.504631 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8157ba61-1f4d-429d-be8d-041654caceb9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:07.505457 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4060e5bd-2a5f-4fd7-a73e-5e41ce65391d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:07.506212 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4060e5bd-2a5f-4fd7-a73e-5e41ce65391d admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:07.506316 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4060e5bd-2a5f-4fd7-a73e-5e41ce65391d admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:07.506613 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4060e5bd-2a5f-4fd7-a73e-5e41ce65391d admin admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:07.507253 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4060e5bd-2a5f-4fd7-a73e-5e41ce65391d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 09:01:07.507626 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4060e5bd-2a5f-4fd7-a73e-5e41ce65391d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:07.509270 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4060e5bd-2a5f-4fd7-a73e-5e41ce65391d admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:07.514534 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8157ba61-1f4d-429d-be8d-041654caceb9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:07.516091 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8157ba61-1f4d-429d-be8d-041654caceb9 service neutron] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:07.521941 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 765/1517] 10.222.0.36 () {58 vars in 1153 bytes} [Thu Oct 20 09:01:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 09:01:07.534905 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4060e5bd-2a5f-4fd7-a73e-5e41ce65391d admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:07.535019 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 753/1518] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:01:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:01:07.535019 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8157ba61-1f4d-429d-be8d-041654caceb9 service neutron] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:07.538895 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4060e5bd-2a5f-4fd7-a73e-5e41ce65391d admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:07.541850 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-081ae4ca-0a4f-4ee2-bebf-e72bc6d729cb tempest-VolumeQuotaClassesTest-1435933842 tempest-VolumeQuotaClassesTest-1435933842-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ee807b6eec054431a863a972ce8ad67b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '624b56c30c8e47d392e621a569ea0bbb', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:07.542498 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-081ae4ca-0a4f-4ee2-bebf-e72bc6d729cb tempest-VolumeQuotaClassesTest-1435933842 tempest-VolumeQuotaClassesTest-1435933842-project-admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:07.542790 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-081ae4ca-0a4f-4ee2-bebf-e72bc6d729cb tempest-VolumeQuotaClassesTest-1435933842 tempest-VolumeQuotaClassesTest-1435933842-project-admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:07.543066 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-081ae4ca-0a4f-4ee2-bebf-e72bc6d729cb tempest-VolumeQuotaClassesTest-1435933842 tempest-VolumeQuotaClassesTest-1435933842-project-admin] PATH_INFO: `/v3/projects` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:07.543699 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-081ae4ca-0a4f-4ee2-bebf-e72bc6d729cb tempest-VolumeQuotaClassesTest-1435933842 tempest-VolumeQuotaClassesTest-1435933842-project-admin] RBAC: Authorizing `identity:create_project()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:07.544161 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8157ba61-1f4d-429d-be8d-041654caceb9 service neutron] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:07.545614 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-081ae4ca-0a4f-4ee2-bebf-e72bc6d729cb tempest-VolumeQuotaClassesTest-1435933842 tempest-VolumeQuotaClassesTest-1435933842-project-admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:07.549027 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc61d6d9-c0a9-4c4b-9ff6-439db1ce2a82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:07.549757 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc61d6d9-c0a9-4c4b-9ff6-439db1ce2a82 admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:07.550197 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc61d6d9-c0a9-4c4b-9ff6-439db1ce2a82 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:07.550566 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc61d6d9-c0a9-4c4b-9ff6-439db1ce2a82 admin admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:07.551336 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc61d6d9-c0a9-4c4b-9ff6-439db1ce2a82 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:07.554056 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc61d6d9-c0a9-4c4b-9ff6-439db1ce2a82 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:07.576174 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 766/1519] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:01:07 2022] POST /identity/v3/projects => generated 353 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:01:07.580160 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-081ae4ca-0a4f-4ee2-bebf-e72bc6d729cb tempest-VolumeQuotaClassesTest-1435933842 tempest-VolumeQuotaClassesTest-1435933842-project-admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:07.587334 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 754/1520] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:01:07 2022] POST /identity/v3/projects => generated 369 bytes in 59 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:01:07.591459 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc61d6d9-c0a9-4c4b-9ff6-439db1ce2a82 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:07.598288 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=89,keystone:DELETE=9,keystone:INSERT=2 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:07.658325 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-081ae4ca-0a4f-4ee2-bebf-e72bc6d729cb tempest-VolumeQuotaClassesTest-1435933842 tempest-VolumeQuotaClassesTest-1435933842-project-admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:07.669834 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc61d6d9-c0a9-4c4b-9ff6-439db1ce2a82 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:07.677742 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75b6cfa9-b0e5-44e4-b37c-5056b87d643f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:07.678463 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75b6cfa9-b0e5-44e4-b37c-5056b87d643f admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:07.678840 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75b6cfa9-b0e5-44e4-b37c-5056b87d643f admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:07.679180 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75b6cfa9-b0e5-44e4-b37c-5056b87d643f admin admin] PATH_INFO: `/v3/users` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:07.681839 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75b6cfa9-b0e5-44e4-b37c-5056b87d643f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:07.685790 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75b6cfa9-b0e5-44e4-b37c-5056b87d643f admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:07.720921 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 755/1521] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:01:07 2022] POST /identity/v3/users => generated 285 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 20 09:01:07.725408 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75b6cfa9-b0e5-44e4-b37c-5056b87d643f admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:07.732644 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b383553f-6d87-4a06-98fa-de8cc248cf81 tempest-VolumeQuotaClassesTest-1435933842 tempest-VolumeQuotaClassesTest-1435933842-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ee807b6eec054431a863a972ce8ad67b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '624b56c30c8e47d392e621a569ea0bbb', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:07.733671 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b383553f-6d87-4a06-98fa-de8cc248cf81 tempest-VolumeQuotaClassesTest-1435933842 tempest-VolumeQuotaClassesTest-1435933842-project-admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:07.734127 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b383553f-6d87-4a06-98fa-de8cc248cf81 tempest-VolumeQuotaClassesTest-1435933842 tempest-VolumeQuotaClassesTest-1435933842-project-admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:07.734600 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b383553f-6d87-4a06-98fa-de8cc248cf81 tempest-VolumeQuotaClassesTest-1435933842 tempest-VolumeQuotaClassesTest-1435933842-project-admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:07.735540 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b383553f-6d87-4a06-98fa-de8cc248cf81 tempest-VolumeQuotaClassesTest-1435933842 tempest-VolumeQuotaClassesTest-1435933842-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 09:01:07.736019 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b383553f-6d87-4a06-98fa-de8cc248cf81 tempest-VolumeQuotaClassesTest-1435933842 tempest-VolumeQuotaClassesTest-1435933842-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:07.737973 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b383553f-6d87-4a06-98fa-de8cc248cf81 tempest-VolumeQuotaClassesTest-1435933842 tempest-VolumeQuotaClassesTest-1435933842-project-admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:07.740244 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75b6cfa9-b0e5-44e4-b37c-5056b87d643f admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:07.746803 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70b34756-e27d-4427-9088-8a8893b5d3a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:07.747697 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70b34756-e27d-4427-9088-8a8893b5d3a8 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:07.748045 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70b34756-e27d-4427-9088-8a8893b5d3a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:07.748386 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70b34756-e27d-4427-9088-8a8893b5d3a8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:07.749394 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70b34756-e27d-4427-9088-8a8893b5d3a8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:07.752712 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70b34756-e27d-4427-9088-8a8893b5d3a8 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:07.762077 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 767/1522] 10.222.0.36 () {58 vars in 1153 bytes} [Thu Oct 20 09:01:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 09:01:07.769193 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 756/1523] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:01:07 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:01:07.770607 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b383553f-6d87-4a06-98fa-de8cc248cf81 tempest-VolumeQuotaClassesTest-1435933842 tempest-VolumeQuotaClassesTest-1435933842-project-admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:07.783539 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b383553f-6d87-4a06-98fa-de8cc248cf81 tempest-VolumeQuotaClassesTest-1435933842 tempest-VolumeQuotaClassesTest-1435933842-project-admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:07.791726 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b7e409-047a-413d-bc0f-adbbdd7f8728 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:07.792341 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b7e409-047a-413d-bc0f-adbbdd7f8728 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:07.792618 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b7e409-047a-413d-bc0f-adbbdd7f8728 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:07.797646 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b7e409-047a-413d-bc0f-adbbdd7f8728 admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:07.798427 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13b7e409-047a-413d-bc0f-adbbdd7f8728 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:07.800072 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13b7e409-047a-413d-bc0f-adbbdd7f8728 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:07.814015 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70b34756-e27d-4427-9088-8a8893b5d3a8 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:07.825218 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70b34756-e27d-4427-9088-8a8893b5d3a8 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:07.828901 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 768/1524] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:01:07 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:01:07.829985 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b7e409-047a-413d-bc0f-adbbdd7f8728 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:07.833914 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca401540-0def-4cd4-bcba-29285a909b6d tempest-VolumeQuotaClassesTest-1435933842 tempest-VolumeQuotaClassesTest-1435933842-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ee807b6eec054431a863a972ce8ad67b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '624b56c30c8e47d392e621a569ea0bbb', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:07.834839 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca401540-0def-4cd4-bcba-29285a909b6d tempest-VolumeQuotaClassesTest-1435933842 tempest-VolumeQuotaClassesTest-1435933842-project-admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:07.835301 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca401540-0def-4cd4-bcba-29285a909b6d tempest-VolumeQuotaClassesTest-1435933842 tempest-VolumeQuotaClassesTest-1435933842-project-admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:07.835750 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca401540-0def-4cd4-bcba-29285a909b6d tempest-VolumeQuotaClassesTest-1435933842 tempest-VolumeQuotaClassesTest-1435933842-project-admin] PATH_INFO: `/v3/projects/b35f84f4ead448cdbadc44eb7183c63a` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:07.838744 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca401540-0def-4cd4-bcba-29285a909b6d tempest-VolumeQuotaClassesTest-1435933842 tempest-VolumeQuotaClassesTest-1435933842-project-admin] RBAC: Authorizing `identity:delete_project(project_id=b35f84f4ead448cdbadc44eb7183c63a)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:07.841836 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca401540-0def-4cd4-bcba-29285a909b6d tempest-VolumeQuotaClassesTest-1435933842 tempest-VolumeQuotaClassesTest-1435933842-project-admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:07.844262 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b7e409-047a-413d-bc0f-adbbdd7f8728 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:07.851033 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca737f94-18df-4fc6-9d7a-d765532b5384 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:07.852784 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca737f94-18df-4fc6-9d7a-d765532b5384 admin admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:07.853274 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca737f94-18df-4fc6-9d7a-d765532b5384 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:07.853728 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca737f94-18df-4fc6-9d7a-d765532b5384 admin admin] PATH_INFO: `/v3/projects/a91bb0a442ec4534ba30082b7be8dbf5/users/e95df59aa56349558136c7e5f4858427/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:07.875998 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca737f94-18df-4fc6-9d7a-d765532b5384 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a91bb0a442ec4534ba30082b7be8dbf5, user_id=e95df59aa56349558136c7e5f4858427, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:07.879529 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca737f94-18df-4fc6-9d7a-d765532b5384 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:07.882240 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:07.910848 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 769/1525] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:01:07 2022] PUT /identity/v3/projects/a91bb0a442ec4534ba30082b7be8dbf5/users/e95df59aa56349558136c7e5f4858427/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 81 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:01:07.925884 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34444f5e-515e-4cc4-8c15-916390106d8c None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:07.925884 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34444f5e-515e-4cc4-8c15-916390106d8c None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:07.926172 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-ca401540-0def-4cd4-bcba-29285a909b6d tempest-VolumeQuotaClassesTest-1435933842 tempest-VolumeQuotaClassesTest-1435933842-project-admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b35f84f4ead448cdbadc44eb7183c63a', 'request_id': 'req-ca401540-0def-4cd4-bcba-29285a909b6d'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:07.926683 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34444f5e-515e-4cc4-8c15-916390106d8c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:07.941999 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-ca401540-0def-4cd4-bcba-29285a909b6d tempest-VolumeQuotaClassesTest-1435933842 tempest-VolumeQuotaClassesTest-1435933842-project-admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b35f84f4ead448cdbadc44eb7183c63a', 'request_id': 'req-ca401540-0def-4cd4-bcba-29285a909b6d'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:07.951643 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-ca401540-0def-4cd4-bcba-29285a909b6d tempest-VolumeQuotaClassesTest-1435933842 tempest-VolumeQuotaClassesTest-1435933842-project-admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b35f84f4ead448cdbadc44eb7183c63a', 'request_id': 'req-ca401540-0def-4cd4-bcba-29285a909b6d'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:07.977470 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-ca401540-0def-4cd4-bcba-29285a909b6d tempest-VolumeQuotaClassesTest-1435933842 tempest-VolumeQuotaClassesTest-1435933842-project-admin] The token cache is being invalidate because project b35f84f4ead448cdbadc44eb7183c63a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81561) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:01:07.977840 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-ca401540-0def-4cd4-bcba-29285a909b6d tempest-VolumeQuotaClassesTest-1435933842 tempest-VolumeQuotaClassesTest-1435933842-project-admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:07.980176 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-ca401540-0def-4cd4-bcba-29285a909b6d tempest-VolumeQuotaClassesTest-1435933842 tempest-VolumeQuotaClassesTest-1435933842-project-admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:07.981925 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 757/1526] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:01:07 2022] DELETE /identity/v3/projects/b35f84f4ead448cdbadc44eb7183c63a => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:01:07.982862 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca401540-0def-4cd4-bcba-29285a909b6d tempest-VolumeQuotaClassesTest-1435933842 tempest-VolumeQuotaClassesTest-1435933842-project-admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:07.984413 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-ca401540-0def-4cd4-bcba-29285a909b6d tempest-VolumeQuotaClassesTest-1435933842 tempest-VolumeQuotaClassesTest-1435933842-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:08.076632 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca401540-0def-4cd4-bcba-29285a909b6d tempest-VolumeQuotaClassesTest-1435933842 tempest-VolumeQuotaClassesTest-1435933842-project-admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:08.084670 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed7ce0eb-cc0f-43fb-bbf3-47f84dbe76a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:08.085583 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed7ce0eb-cc0f-43fb-bbf3-47f84dbe76a9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:08.086584 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed7ce0eb-cc0f-43fb-bbf3-47f84dbe76a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:08.087049 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed7ce0eb-cc0f-43fb-bbf3-47f84dbe76a9 admin admin] PATH_INFO: `/v3/projects/fc728a8b82294937b1dcc1edb6528e36` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:08.088977 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed7ce0eb-cc0f-43fb-bbf3-47f84dbe76a9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fc728a8b82294937b1dcc1edb6528e36)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:08.094899 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed7ce0eb-cc0f-43fb-bbf3-47f84dbe76a9 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:08.103495 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-34444f5e-515e-4cc4-8c15-916390106d8c None None] MFA Rules not processed for user `e95df59aa56349558136c7e5f4858427`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:08.106150 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:08.106507 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-34444f5e-515e-4cc4-8c15-916390106d8c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:08.194516 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 770/1527] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:07 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 271 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:08.199343 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-ed7ce0eb-cc0f-43fb-bbf3-47f84dbe76a9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fc728a8b82294937b1dcc1edb6528e36', 'request_id': 'req-ed7ce0eb-cc0f-43fb-bbf3-47f84dbe76a9'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:08.206407 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-ed7ce0eb-cc0f-43fb-bbf3-47f84dbe76a9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fc728a8b82294937b1dcc1edb6528e36', 'request_id': 'req-ed7ce0eb-cc0f-43fb-bbf3-47f84dbe76a9'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:08.209027 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34444f5e-515e-4cc4-8c15-916390106d8c None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:08.210626 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-34444f5e-515e-4cc4-8c15-916390106d8c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:08.217888 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-ed7ce0eb-cc0f-43fb-bbf3-47f84dbe76a9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fc728a8b82294937b1dcc1edb6528e36', 'request_id': 'req-ed7ce0eb-cc0f-43fb-bbf3-47f84dbe76a9'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:08.244145 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-ed7ce0eb-cc0f-43fb-bbf3-47f84dbe76a9 admin admin] The token cache is being invalidate because project fc728a8b82294937b1dcc1edb6528e36 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81561) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:01:08.245943 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-ed7ce0eb-cc0f-43fb-bbf3-47f84dbe76a9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:08.247931 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-ed7ce0eb-cc0f-43fb-bbf3-47f84dbe76a9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:08.252610 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 758/1528] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:01:07 2022] DELETE /identity/v3/projects/fc728a8b82294937b1dcc1edb6528e36 => generated 0 bytes in 270 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:08.256570 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed7ce0eb-cc0f-43fb-bbf3-47f84dbe76a9 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:08.258747 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-ed7ce0eb-cc0f-43fb-bbf3-47f84dbe76a9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:08.311211 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34444f5e-515e-4cc4-8c15-916390106d8c None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:08.332015 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed7ce0eb-cc0f-43fb-bbf3-47f84dbe76a9 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:08.337866 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed70ba75-5d13-4bc1-bb99-106bf5905eba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:08.338147 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed70ba75-5d13-4bc1-bb99-106bf5905eba admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:08.338423 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed70ba75-5d13-4bc1-bb99-106bf5905eba admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:08.338733 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed70ba75-5d13-4bc1-bb99-106bf5905eba admin admin] PATH_INFO: `/v3/users/978ebf3bc10d45d5bb251ebd10b28a29` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:08.340377 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed70ba75-5d13-4bc1-bb99-106bf5905eba admin admin] RBAC: Authorizing `identity:delete_user(user_id=978ebf3bc10d45d5bb251ebd10b28a29)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:08.342408 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed70ba75-5d13-4bc1-bb99-106bf5905eba admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:08.373667 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34cbec61-e313-4bac-85f4-ad13961e8853 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce043914cd9042909f70ab0adb272d34', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:08.374417 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34cbec61-e313-4bac-85f4-ad13961e8853 service glance] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:08.374771 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34cbec61-e313-4bac-85f4-ad13961e8853 service glance] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:08.378773 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34cbec61-e313-4bac-85f4-ad13961e8853 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:08.380829 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-34cbec61-e313-4bac-85f4-ad13961e8853 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:08.399954 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-ed70ba75-5d13-4bc1-bb99-106bf5905eba admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '978ebf3bc10d45d5bb251ebd10b28a29'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:08.408827 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-ed70ba75-5d13-4bc1-bb99-106bf5905eba admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '978ebf3bc10d45d5bb251ebd10b28a29'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:08.422791 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-ed70ba75-5d13-4bc1-bb99-106bf5905eba admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '978ebf3bc10d45d5bb251ebd10b28a29'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:08.423040 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:08.431108 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-ed70ba75-5d13-4bc1-bb99-106bf5905eba admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '978ebf3bc10d45d5bb251ebd10b28a29'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:08.431809 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-ed70ba75-5d13-4bc1-bb99-106bf5905eba admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '978ebf3bc10d45d5bb251ebd10b28a29'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:08.437709 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 759/1529] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:01:08 2022] DELETE /identity/v3/users/978ebf3bc10d45d5bb251ebd10b28a29 => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:01:08.438707 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed70ba75-5d13-4bc1-bb99-106bf5905eba admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:08.440130 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-ed70ba75-5d13-4bc1-bb99-106bf5905eba admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:08.452779 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34cbec61-e313-4bac-85f4-ad13961e8853 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:08.454252 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34cbec61-e313-4bac-85f4-ad13961e8853 service glance] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:08.461952 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:08.470597 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 771/1530] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:01:08 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 262 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 20 09:01:08.524924 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed70ba75-5d13-4bc1-bb99-106bf5905eba admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:08.530092 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d3cdc83-c0e9-4fda-8a9d-5d5f53deb22d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:08.530844 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d3cdc83-c0e9-4fda-8a9d-5d5f53deb22d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:08.531092 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d3cdc83-c0e9-4fda-8a9d-5d5f53deb22d admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:08.531364 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d3cdc83-c0e9-4fda-8a9d-5d5f53deb22d admin admin] PATH_INFO: `/v3/users/035b8a17c4594f279853c63ed2cd00c5` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:08.532508 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d3cdc83-c0e9-4fda-8a9d-5d5f53deb22d admin admin] RBAC: Authorizing `identity:delete_user(user_id=035b8a17c4594f279853c63ed2cd00c5)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:08.534444 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d3cdc83-c0e9-4fda-8a9d-5d5f53deb22d admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:08.602333 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-3d3cdc83-c0e9-4fda-8a9d-5d5f53deb22d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '035b8a17c4594f279853c63ed2cd00c5'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:08.618937 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-3d3cdc83-c0e9-4fda-8a9d-5d5f53deb22d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '035b8a17c4594f279853c63ed2cd00c5'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:08.640102 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-3d3cdc83-c0e9-4fda-8a9d-5d5f53deb22d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '035b8a17c4594f279853c63ed2cd00c5'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:08.647049 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-3d3cdc83-c0e9-4fda-8a9d-5d5f53deb22d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '035b8a17c4594f279853c63ed2cd00c5'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:08.647905 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-3d3cdc83-c0e9-4fda-8a9d-5d5f53deb22d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '035b8a17c4594f279853c63ed2cd00c5'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:08.654718 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 760/1531] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:01:08 2022] DELETE /identity/v3/users/035b8a17c4594f279853c63ed2cd00c5 => generated 0 bytes in 216 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:08.659077 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-123e4a43-cc07-4838-a628-6dd4fd83dc1e None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:08.659362 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-123e4a43-cc07-4838-a628-6dd4fd83dc1e None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:08.659630 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-123e4a43-cc07-4838-a628-6dd4fd83dc1e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:08.679027 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-123e4a43-cc07-4838-a628-6dd4fd83dc1e None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:08.681478 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-123e4a43-cc07-4838-a628-6dd4fd83dc1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:08.758328 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 772/1532] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:08.768834 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d3cdc83-c0e9-4fda-8a9d-5d5f53deb22d admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:08.770362 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-3d3cdc83-c0e9-4fda-8a9d-5d5f53deb22d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:08.785937 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-123e4a43-cc07-4838-a628-6dd4fd83dc1e None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:08.787390 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-123e4a43-cc07-4838-a628-6dd4fd83dc1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:08.802607 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-123e4a43-cc07-4838-a628-6dd4fd83dc1e None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:08.811049 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f436b6-4a39-481e-9163-2d66277a5c92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:08.811647 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f436b6-4a39-481e-9163-2d66277a5c92 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:08.811925 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f436b6-4a39-481e-9163-2d66277a5c92 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:08.812192 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f436b6-4a39-481e-9163-2d66277a5c92 admin admin] PATH_INFO: `/v3/projects/ce934e25f1ed422abf53d5e384c01518` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:08.815283 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20f436b6-4a39-481e-9163-2d66277a5c92 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ce934e25f1ed422abf53d5e384c01518)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:08.818413 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20f436b6-4a39-481e-9163-2d66277a5c92 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:08.841865 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d3cdc83-c0e9-4fda-8a9d-5d5f53deb22d admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:08.846402 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91b29b8d-0e86-4fe8-af59-f4d50a7b1bc4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:08.847685 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b29b8d-0e86-4fe8-af59-f4d50a7b1bc4 service neutron] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:08.847685 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b29b8d-0e86-4fe8-af59-f4d50a7b1bc4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:08.847892 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b29b8d-0e86-4fe8-af59-f4d50a7b1bc4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:08.859181 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91b29b8d-0e86-4fe8-af59-f4d50a7b1bc4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:08.860696 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91b29b8d-0e86-4fe8-af59-f4d50a7b1bc4 service neutron] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:08.872346 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 761/1533] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:01:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:01:08.878261 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-20f436b6-4a39-481e-9163-2d66277a5c92 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ce934e25f1ed422abf53d5e384c01518', 'request_id': 'req-20f436b6-4a39-481e-9163-2d66277a5c92'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:08.885469 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-20f436b6-4a39-481e-9163-2d66277a5c92 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ce934e25f1ed422abf53d5e384c01518', 'request_id': 'req-20f436b6-4a39-481e-9163-2d66277a5c92'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:08.891779 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-20f436b6-4a39-481e-9163-2d66277a5c92 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ce934e25f1ed422abf53d5e384c01518', 'request_id': 'req-20f436b6-4a39-481e-9163-2d66277a5c92'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:08.915655 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-20f436b6-4a39-481e-9163-2d66277a5c92 admin admin] The token cache is being invalidate because project ce934e25f1ed422abf53d5e384c01518 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81562) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:01:08.916041 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-20f436b6-4a39-481e-9163-2d66277a5c92 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:08.916900 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-20f436b6-4a39-481e-9163-2d66277a5c92 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:08.919402 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 773/1534] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:01:08 2022] DELETE /identity/v3/projects/ce934e25f1ed422abf53d5e384c01518 => generated 0 bytes in 134 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:01:08.924344 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c8d95d-4b38-4480-97c5-8c5f190a4ae1 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:08.924630 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c8d95d-4b38-4480-97c5-8c5f190a4ae1 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:08.924903 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c8d95d-4b38-4480-97c5-8c5f190a4ae1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:08.960017 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-25c8d95d-4b38-4480-97c5-8c5f190a4ae1 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:08.961291 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-25c8d95d-4b38-4480-97c5-8c5f190a4ae1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:09.042755 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:09.089391 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 762/1535] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:09.098916 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1958aaf0-183a-4184-95b3-7b8e59ae1794 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:09.099330 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1958aaf0-183a-4184-95b3-7b8e59ae1794 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:09.099670 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1958aaf0-183a-4184-95b3-7b8e59ae1794 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:09.148607 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-1958aaf0-183a-4184-95b3-7b8e59ae1794 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:09.149903 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-1958aaf0-183a-4184-95b3-7b8e59ae1794 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:09.166743 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 774/1536] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 69 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:09.167691 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25c8d95d-4b38-4480-97c5-8c5f190a4ae1 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:09.184964 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25c8d95d-4b38-4480-97c5-8c5f190a4ae1 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:09.192904 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c5be4bb-b08a-43fc-a06d-469273d04600 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:09.194075 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c5be4bb-b08a-43fc-a06d-469273d04600 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:09.194605 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c5be4bb-b08a-43fc-a06d-469273d04600 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:09.194940 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c5be4bb-b08a-43fc-a06d-469273d04600 admin admin] PATH_INFO: `/v3/domains` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:09.196122 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c5be4bb-b08a-43fc-a06d-469273d04600 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 09:01:09.196122 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c5be4bb-b08a-43fc-a06d-469273d04600 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:09.198458 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c5be4bb-b08a-43fc-a06d-469273d04600 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:09.215678 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 763/1537] 10.222.0.36 () {58 vars in 1153 bytes} [Thu Oct 20 09:01:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 09:01:09.217242 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1958aaf0-183a-4184-95b3-7b8e59ae1794 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:09.228301 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1958aaf0-183a-4184-95b3-7b8e59ae1794 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:09.233835 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da86425-4434-476a-b85b-82ad66be4198 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:09.234761 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da86425-4434-476a-b85b-82ad66be4198 admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:09.235218 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da86425-4434-476a-b85b-82ad66be4198 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:09.235697 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da86425-4434-476a-b85b-82ad66be4198 admin admin] PATH_INFO: `/v3/projects` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:09.236852 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4da86425-4434-476a-b85b-82ad66be4198 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:09.238985 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4da86425-4434-476a-b85b-82ad66be4198 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:09.290515 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 775/1538] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:01:09 2022] POST /identity/v3/projects => generated 387 bytes in 71 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:01:09.292424 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c5be4bb-b08a-43fc-a06d-469273d04600 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:09.303074 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c5be4bb-b08a-43fc-a06d-469273d04600 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:09.306066 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da86425-4434-476a-b85b-82ad66be4198 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:09.307721 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-4da86425-4434-476a-b85b-82ad66be4198 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:09.341558 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:09.397120 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da86425-4434-476a-b85b-82ad66be4198 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:09.398060 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ac5c4f8-8bac-492a-9397-1b01123169cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:09.399532 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac5c4f8-8bac-492a-9397-1b01123169cc admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:09.399986 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac5c4f8-8bac-492a-9397-1b01123169cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:09.400455 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac5c4f8-8bac-492a-9397-1b01123169cc admin admin] PATH_INFO: `/v3/users` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:09.401257 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ac5c4f8-8bac-492a-9397-1b01123169cc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:09.402866 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bec25d99-71d5-4023-bd6f-2c8c510bf4ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:09.403848 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec25d99-71d5-4023-bd6f-2c8c510bf4ee admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:09.404213 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec25d99-71d5-4023-bd6f-2c8c510bf4ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:09.404559 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec25d99-71d5-4023-bd6f-2c8c510bf4ee admin admin] PATH_INFO: `/v3/projects/7a8bb34c215647bc98941e0acd5a6da7` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:09.405914 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ac5c4f8-8bac-492a-9397-1b01123169cc admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:09.406053 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bec25d99-71d5-4023-bd6f-2c8c510bf4ee admin admin] RBAC: Authorizing `identity:delete_project(project_id=7a8bb34c215647bc98941e0acd5a6da7)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:09.408090 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bec25d99-71d5-4023-bd6f-2c8c510bf4ee admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:09.440318 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 764/1539] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:01:09 2022] POST /identity/v3/users => generated 294 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:01:09.444564 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ac5c4f8-8bac-492a-9397-1b01123169cc admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:09.466600 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ac5c4f8-8bac-492a-9397-1b01123169cc admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:09.466600 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cad1e5f-a0bd-4bdf-90d0-c2601fcce057 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:09.466600 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cad1e5f-a0bd-4bdf-90d0-c2601fcce057 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:09.466600 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cad1e5f-a0bd-4bdf-90d0-c2601fcce057 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:09.467226 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cad1e5f-a0bd-4bdf-90d0-c2601fcce057 admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:09.467226 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cad1e5f-a0bd-4bdf-90d0-c2601fcce057 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:09.467226 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cad1e5f-a0bd-4bdf-90d0-c2601fcce057 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:09.483481 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 765/1540] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:01:09 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 20 09:01:09.487551 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cad1e5f-a0bd-4bdf-90d0-c2601fcce057 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:09.491804 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-bec25d99-71d5-4023-bd6f-2c8c510bf4ee admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7a8bb34c215647bc98941e0acd5a6da7', 'request_id': 'req-bec25d99-71d5-4023-bd6f-2c8c510bf4ee'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:09.499431 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cad1e5f-a0bd-4bdf-90d0-c2601fcce057 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:09.499670 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-bec25d99-71d5-4023-bd6f-2c8c510bf4ee admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7a8bb34c215647bc98941e0acd5a6da7', 'request_id': 'req-bec25d99-71d5-4023-bd6f-2c8c510bf4ee'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:09.506173 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-bec25d99-71d5-4023-bd6f-2c8c510bf4ee admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7a8bb34c215647bc98941e0acd5a6da7', 'request_id': 'req-bec25d99-71d5-4023-bd6f-2c8c510bf4ee'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:09.523844 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-bec25d99-71d5-4023-bd6f-2c8c510bf4ee admin admin] The token cache is being invalidate because project 7a8bb34c215647bc98941e0acd5a6da7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81562) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:01:09.524162 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-bec25d99-71d5-4023-bd6f-2c8c510bf4ee admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:09.524790 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-bec25d99-71d5-4023-bd6f-2c8c510bf4ee admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:09.527016 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 776/1541] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:01:09 2022] DELETE /identity/v3/projects/7a8bb34c215647bc98941e0acd5a6da7 => generated 0 bytes in 221 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:01:09.531644 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bec25d99-71d5-4023-bd6f-2c8c510bf4ee admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:09.533306 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-bec25d99-71d5-4023-bd6f-2c8c510bf4ee admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:09.563987 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c9c5f12-800c-4206-a306-a58468428fc4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:09.564600 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9c5f12-800c-4206-a306-a58468428fc4 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:09.564943 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9c5f12-800c-4206-a306-a58468428fc4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:09.565220 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9c5f12-800c-4206-a306-a58468428fc4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:09.565863 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c9c5f12-800c-4206-a306-a58468428fc4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:09.567569 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c9c5f12-800c-4206-a306-a58468428fc4 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:09.578632 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 766/1542] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:01:09 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:01:09.582746 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c9c5f12-800c-4206-a306-a58468428fc4 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:09.584471 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-2c9c5f12-800c-4206-a306-a58468428fc4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:09.598240 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c9c5f12-800c-4206-a306-a58468428fc4 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:09.607860 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f68fdd4-0eeb-4bb4-b7b1-ebb8682e3f9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:09.608598 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f68fdd4-0eeb-4bb4-b7b1-ebb8682e3f9d admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:09.608943 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f68fdd4-0eeb-4bb4-b7b1-ebb8682e3f9d admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:09.609219 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f68fdd4-0eeb-4bb4-b7b1-ebb8682e3f9d admin admin] PATH_INFO: `/v3/projects/7b1ffd8e666743fdbc24a4cde3a37192/users/598f0b30f9d34057bb75d6131137ed68/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:09.613064 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bec25d99-71d5-4023-bd6f-2c8c510bf4ee admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:09.618040 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81afc04a-f659-458f-ae94-8b78854699da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:09.618663 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81afc04a-f659-458f-ae94-8b78854699da admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:09.619126 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81afc04a-f659-458f-ae94-8b78854699da admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:09.619473 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81afc04a-f659-458f-ae94-8b78854699da admin admin] PATH_INFO: `/v3/users/ee807b6eec054431a863a972ce8ad67b` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:09.620708 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81afc04a-f659-458f-ae94-8b78854699da admin admin] RBAC: Authorizing `identity:delete_user(user_id=ee807b6eec054431a863a972ce8ad67b)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:09.622837 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81afc04a-f659-458f-ae94-8b78854699da admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:09.624861 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f68fdd4-0eeb-4bb4-b7b1-ebb8682e3f9d admin admin] RBAC: Authorizing `identity:create_grant(project_id=7b1ffd8e666743fdbc24a4cde3a37192, user_id=598f0b30f9d34057bb75d6131137ed68, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:09.627565 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f68fdd4-0eeb-4bb4-b7b1-ebb8682e3f9d admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:09.645451 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 767/1543] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:01:09 2022] PUT /identity/v3/projects/7b1ffd8e666743fdbc24a4cde3a37192/users/598f0b30f9d34057bb75d6131137ed68/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:01:09.658861 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b36d0c87-b5bb-4402-bd5a-5d9241e419c9 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:09.659153 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b36d0c87-b5bb-4402-bd5a-5d9241e419c9 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:09.659441 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b36d0c87-b5bb-4402-bd5a-5d9241e419c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:09.697816 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-81afc04a-f659-458f-ae94-8b78854699da admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ee807b6eec054431a863a972ce8ad67b'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:09.704168 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-b36d0c87-b5bb-4402-bd5a-5d9241e419c9 None None] MFA Rules not processed for user `598f0b30f9d34057bb75d6131137ed68`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:09.706709 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-b36d0c87-b5bb-4402-bd5a-5d9241e419c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:09.707578 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-81afc04a-f659-458f-ae94-8b78854699da admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ee807b6eec054431a863a972ce8ad67b'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:09.713958 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:09.718837 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=82,keystone:DELETE=16,keystone:INSERT=2 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:09.733550 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-81afc04a-f659-458f-ae94-8b78854699da admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ee807b6eec054431a863a972ce8ad67b'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:09.747093 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-81afc04a-f659-458f-ae94-8b78854699da admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ee807b6eec054431a863a972ce8ad67b'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:09.747928 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-81afc04a-f659-458f-ae94-8b78854699da admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ee807b6eec054431a863a972ce8ad67b'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:09.754835 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 777/1544] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:01:09 2022] DELETE /identity/v3/users/ee807b6eec054431a863a972ce8ad67b => generated 0 bytes in 224 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:09.798761 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 768/1545] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:09 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 20 09:01:09.803714 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81afc04a-f659-458f-ae94-8b78854699da admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:09.805774 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-81afc04a-f659-458f-ae94-8b78854699da admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:09.879234 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81afc04a-f659-458f-ae94-8b78854699da admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:09.884012 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d97ebbe7-1e2f-416b-80bb-1426e11e1dd0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:09.884625 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d97ebbe7-1e2f-416b-80bb-1426e11e1dd0 admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:09.884885 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d97ebbe7-1e2f-416b-80bb-1426e11e1dd0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:09.885149 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d97ebbe7-1e2f-416b-80bb-1426e11e1dd0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:09.885808 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d97ebbe7-1e2f-416b-80bb-1426e11e1dd0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:09.887748 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d97ebbe7-1e2f-416b-80bb-1426e11e1dd0 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:09.921697 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 778/1546] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:01:09 2022] POST /identity/v3/projects => generated 389 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 20 09:01:09.925995 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b36d0c87-b5bb-4402-bd5a-5d9241e419c9 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:09.998022 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b36d0c87-b5bb-4402-bd5a-5d9241e419c9 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:10.058588 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d97ebbe7-1e2f-416b-80bb-1426e11e1dd0 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:10.060699 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-d97ebbe7-1e2f-416b-80bb-1426e11e1dd0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:10.064596 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb5b49b3-b52f-4c1c-8dbf-6b0dd65735aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:10.065389 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5b49b3-b52f-4c1c-8dbf-6b0dd65735aa admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:10.065678 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5b49b3-b52f-4c1c-8dbf-6b0dd65735aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:10.065948 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5b49b3-b52f-4c1c-8dbf-6b0dd65735aa admin admin] PATH_INFO: `/v3/users` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:10.066621 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb5b49b3-b52f-4c1c-8dbf-6b0dd65735aa admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:10.069317 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb5b49b3-b52f-4c1c-8dbf-6b0dd65735aa admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:10.083733 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d97ebbe7-1e2f-416b-80bb-1426e11e1dd0 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:10.090743 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-420ac954-9a77-4a7e-9ce3-f8807c557315 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:10.091471 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-420ac954-9a77-4a7e-9ce3-f8807c557315 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:10.091754 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-420ac954-9a77-4a7e-9ce3-f8807c557315 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:10.092030 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-420ac954-9a77-4a7e-9ce3-f8807c557315 admin admin] PATH_INFO: `/v3/projects/624b56c30c8e47d392e621a569ea0bbb` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:10.093228 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-420ac954-9a77-4a7e-9ce3-f8807c557315 admin admin] RBAC: Authorizing `identity:delete_project(project_id=624b56c30c8e47d392e621a569ea0bbb)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:10.095212 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-420ac954-9a77-4a7e-9ce3-f8807c557315 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:10.102812 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 769/1547] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:01:09 2022] POST /identity/v3/users => generated 301 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:01:10.105089 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb5b49b3-b52f-4c1c-8dbf-6b0dd65735aa admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:10.116659 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb5b49b3-b52f-4c1c-8dbf-6b0dd65735aa admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:10.121391 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22f253f8-0caa-4a74-b669-30c36b9ad99a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:10.122006 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f253f8-0caa-4a74-b669-30c36b9ad99a admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:10.122283 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f253f8-0caa-4a74-b669-30c36b9ad99a admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:10.122972 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f253f8-0caa-4a74-b669-30c36b9ad99a admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:10.123625 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22f253f8-0caa-4a74-b669-30c36b9ad99a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:10.125426 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22f253f8-0caa-4a74-b669-30c36b9ad99a admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:10.147432 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 770/1548] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:01:10 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:01:10.149454 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22f253f8-0caa-4a74-b669-30c36b9ad99a admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:10.158288 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-420ac954-9a77-4a7e-9ce3-f8807c557315 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '624b56c30c8e47d392e621a569ea0bbb', 'request_id': 'req-420ac954-9a77-4a7e-9ce3-f8807c557315'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:10.171488 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22f253f8-0caa-4a74-b669-30c36b9ad99a admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:10.172726 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-420ac954-9a77-4a7e-9ce3-f8807c557315 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '624b56c30c8e47d392e621a569ea0bbb', 'request_id': 'req-420ac954-9a77-4a7e-9ce3-f8807c557315'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:10.180364 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-420ac954-9a77-4a7e-9ce3-f8807c557315 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '624b56c30c8e47d392e621a569ea0bbb', 'request_id': 'req-420ac954-9a77-4a7e-9ce3-f8807c557315'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:10.207919 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-420ac954-9a77-4a7e-9ce3-f8807c557315 admin admin] The token cache is being invalidate because project 624b56c30c8e47d392e621a569ea0bbb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81562) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:01:10.208426 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-420ac954-9a77-4a7e-9ce3-f8807c557315 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:10.209173 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-420ac954-9a77-4a7e-9ce3-f8807c557315 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:10.214372 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 779/1549] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:01:10 2022] DELETE /identity/v3/projects/624b56c30c8e47d392e621a569ea0bbb => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:10.219741 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b55047b0-daa9-418f-af25-bae173b4f874 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:10.220078 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b55047b0-daa9-418f-af25-bae173b4f874 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:10.220783 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b55047b0-daa9-418f-af25-bae173b4f874 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:10.231942 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:10.253276 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-b55047b0-daa9-418f-af25-bae173b4f874 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:10.254973 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-b55047b0-daa9-418f-af25-bae173b4f874 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:10.280464 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ed4540-1dee-4fa1-ba44-fb9b7de4bc46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:10.281213 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ed4540-1dee-4fa1-ba44-fb9b7de4bc46 admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:10.281502 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ed4540-1dee-4fa1-ba44-fb9b7de4bc46 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:10.281826 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ed4540-1dee-4fa1-ba44-fb9b7de4bc46 admin admin] PATH_INFO: `/v3/projects/2979e4ec6ee54329a498477be56e7a72/users/2f12b6c0f67e44fcacd6a3b834a315de/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:10.302641 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66ed4540-1dee-4fa1-ba44-fb9b7de4bc46 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2979e4ec6ee54329a498477be56e7a72, user_id=2f12b6c0f67e44fcacd6a3b834a315de, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:10.306254 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=90,keystone:DELETE=8,keystone:INSERT=2 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:10.308880 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66ed4540-1dee-4fa1-ba44-fb9b7de4bc46 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:10.335480 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 780/1550] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:10.343848 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f8c95d-83af-4aad-af64-819b57bf124c None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:10.344216 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f8c95d-83af-4aad-af64-819b57bf124c None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:10.344661 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f8c95d-83af-4aad-af64-819b57bf124c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:10.352960 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 771/1551] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:01:10 2022] PUT /identity/v3/projects/2979e4ec6ee54329a498477be56e7a72/users/2f12b6c0f67e44fcacd6a3b834a315de/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:10.354101 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ed4540-1dee-4fa1-ba44-fb9b7de4bc46 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:10.356503 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-66ed4540-1dee-4fa1-ba44-fb9b7de4bc46 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:10.380807 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-38f8c95d-83af-4aad-af64-819b57bf124c None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:10.382235 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-38f8c95d-83af-4aad-af64-819b57bf124c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:10.464775 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ed4540-1dee-4fa1-ba44-fb9b7de4bc46 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:10.471716 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f480505-3e41-45f6-80f3-5eab6b7db971 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:10.472351 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f480505-3e41-45f6-80f3-5eab6b7db971 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:10.472715 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f480505-3e41-45f6-80f3-5eab6b7db971 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:10.473061 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f480505-3e41-45f6-80f3-5eab6b7db971 admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:10.473780 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f480505-3e41-45f6-80f3-5eab6b7db971 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:10.475459 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f480505-3e41-45f6-80f3-5eab6b7db971 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:10.488244 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 781/1552] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:10.492815 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38f8c95d-83af-4aad-af64-819b57bf124c None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:10.499672 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 772/1553] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:01:10 2022] GET /identity/v3/roles => generated 1420 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:01:10.501425 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f480505-3e41-45f6-80f3-5eab6b7db971 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:10.509871 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38f8c95d-83af-4aad-af64-819b57bf124c None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:10.515318 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f480505-3e41-45f6-80f3-5eab6b7db971 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:10.515961 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3004a33f-1313-4f92-b8d8-c745010c870c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:10.516773 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3004a33f-1313-4f92-b8d8-c745010c870c admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:10.517378 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3004a33f-1313-4f92-b8d8-c745010c870c admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:10.517837 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3004a33f-1313-4f92-b8d8-c745010c870c admin admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:10.519102 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3004a33f-1313-4f92-b8d8-c745010c870c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 09:01:10.519941 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3004a33f-1313-4f92-b8d8-c745010c870c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:10.526500 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3004a33f-1313-4f92-b8d8-c745010c870c admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:10.527710 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53b71778-c340-4ca0-b892-6c288cd56fae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:10.528587 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53b71778-c340-4ca0-b892-6c288cd56fae admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:10.529045 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53b71778-c340-4ca0-b892-6c288cd56fae admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:10.529518 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53b71778-c340-4ca0-b892-6c288cd56fae admin admin] PATH_INFO: `/v3/projects/2979e4ec6ee54329a498477be56e7a72/users/2f12b6c0f67e44fcacd6a3b834a315de/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:10.532971 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53b71778-c340-4ca0-b892-6c288cd56fae admin admin] RBAC: Authorizing `identity:create_grant(project_id=2979e4ec6ee54329a498477be56e7a72, user_id=2f12b6c0f67e44fcacd6a3b834a315de, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:10.535953 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53b71778-c340-4ca0-b892-6c288cd56fae admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:10.552819 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 773/1554] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:01:10 2022] PUT /identity/v3/projects/2979e4ec6ee54329a498477be56e7a72/users/2f12b6c0f67e44fcacd6a3b834a315de/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:01:10.555523 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 782/1555] 10.222.0.36 () {58 vars in 1153 bytes} [Thu Oct 20 09:01:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 64 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 09:01:10.560368 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53b71778-c340-4ca0-b892-6c288cd56fae admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:10.564940 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edda6bee-2ec8-4f26-aee1-586a72ac60c8 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:10.565296 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edda6bee-2ec8-4f26-aee1-586a72ac60c8 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:10.565767 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edda6bee-2ec8-4f26-aee1-586a72ac60c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:10.598669 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-edda6bee-2ec8-4f26-aee1-586a72ac60c8 None None] MFA Rules not processed for user `2f12b6c0f67e44fcacd6a3b834a315de`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:10.599658 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-edda6bee-2ec8-4f26-aee1-586a72ac60c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:10.637872 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53b71778-c340-4ca0-b892-6c288cd56fae admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:10.642758 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86838c1f-5afa-4b74-b794-bd931f0c26ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:10.644197 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86838c1f-5afa-4b74-b794-bd931f0c26ce admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:10.644737 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86838c1f-5afa-4b74-b794-bd931f0c26ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:10.645024 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86838c1f-5afa-4b74-b794-bd931f0c26ce admin admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:10.646024 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86838c1f-5afa-4b74-b794-bd931f0c26ce admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:10.651680 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86838c1f-5afa-4b74-b794-bd931f0c26ce admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:10.691455 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 783/1556] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:10 2022] POST /identity/v3/auth/tokens => generated 3333 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:10.693141 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 774/1557] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:01:10 2022] POST /identity/v3/projects => generated 365 bytes in 133 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 20 09:01:10.697412 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edda6bee-2ec8-4f26-aee1-586a72ac60c8 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:10.704624 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86838c1f-5afa-4b74-b794-bd931f0c26ce admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:10.707729 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-86838c1f-5afa-4b74-b794-bd931f0c26ce admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:10.709028 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edda6bee-2ec8-4f26-aee1-586a72ac60c8 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:10.742058 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:10.779941 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f70f78-dbb5-45c7-8706-0ff6e3bd8c54 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:10.780564 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f70f78-dbb5-45c7-8706-0ff6e3bd8c54 admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:10.780956 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f70f78-dbb5-45c7-8706-0ff6e3bd8c54 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:10.781233 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f70f78-dbb5-45c7-8706-0ff6e3bd8c54 admin admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:10.781867 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6f70f78-dbb5-45c7-8706-0ff6e3bd8c54 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:10.783973 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6f70f78-dbb5-45c7-8706-0ff6e3bd8c54 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:10.790953 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86838c1f-5afa-4b74-b794-bd931f0c26ce admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:10.794574 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:10.797254 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc5620be-9d5c-4cf2-99f0-a5dff864940b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:10.798629 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc5620be-9d5c-4cf2-99f0-a5dff864940b service cinder] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:10.799206 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc5620be-9d5c-4cf2-99f0-a5dff864940b service cinder] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:10.799312 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc5620be-9d5c-4cf2-99f0-a5dff864940b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:10.823073 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 784/1558] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:01:10 2022] POST /identity/v3/users => generated 283 bytes in 123 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:01:10.824761 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f70f78-dbb5-45c7-8706-0ff6e3bd8c54 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:10.835497 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f70f78-dbb5-45c7-8706-0ff6e3bd8c54 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:10.845923 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-915c95cf-af6e-4994-9837-2e64f1423207 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:10.846591 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-915c95cf-af6e-4994-9837-2e64f1423207 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:10.846875 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-915c95cf-af6e-4994-9837-2e64f1423207 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:10.847160 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-915c95cf-af6e-4994-9837-2e64f1423207 admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:10.847795 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-915c95cf-af6e-4994-9837-2e64f1423207 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:10.849582 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-915c95cf-af6e-4994-9837-2e64f1423207 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:10.866691 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 785/1559] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:01:10 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:01:10.869372 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-915c95cf-af6e-4994-9837-2e64f1423207 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:10.890539 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-915c95cf-af6e-4994-9837-2e64f1423207 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:10.900500 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56157efc-e7f2-45eb-b5df-a4483d846867 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:10.901141 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56157efc-e7f2-45eb-b5df-a4483d846867 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:10.901418 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56157efc-e7f2-45eb-b5df-a4483d846867 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:10.901740 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56157efc-e7f2-45eb-b5df-a4483d846867 admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:10.902389 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56157efc-e7f2-45eb-b5df-a4483d846867 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:10.904045 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56157efc-e7f2-45eb-b5df-a4483d846867 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:10.906861 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc5620be-9d5c-4cf2-99f0-a5dff864940b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:10.911232 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc5620be-9d5c-4cf2-99f0-a5dff864940b service cinder] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:10.921815 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 786/1560] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:01:10 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 20 09:01:10.922378 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 775/1561] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:01:10 2022] GET /identity/v3/auth/tokens => generated 3333 bytes in 218 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:01:10.926375 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56157efc-e7f2-45eb-b5df-a4483d846867 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:10.947416 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56157efc-e7f2-45eb-b5df-a4483d846867 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:10.953451 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-124de92b-75cc-43b0-a0d7-fa6c81574f4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:10.954313 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-124de92b-75cc-43b0-a0d7-fa6c81574f4d admin admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:10.954671 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-124de92b-75cc-43b0-a0d7-fa6c81574f4d admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:10.954942 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-124de92b-75cc-43b0-a0d7-fa6c81574f4d admin admin] PATH_INFO: `/v3/projects/b40b02ee60bd47f9b2f3352911fe6aa4/users/c185abe69a6e452dbfac0c057e64d369/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:10.975494 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-124de92b-75cc-43b0-a0d7-fa6c81574f4d admin admin] RBAC: Authorizing `identity:create_grant(project_id=b40b02ee60bd47f9b2f3352911fe6aa4, user_id=c185abe69a6e452dbfac0c057e64d369, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:10.978704 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-124de92b-75cc-43b0-a0d7-fa6c81574f4d admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:11.000537 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 787/1562] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:01:10 2022] PUT /identity/v3/projects/b40b02ee60bd47f9b2f3352911fe6aa4/users/c185abe69a6e452dbfac0c057e64d369/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 77 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:11.011824 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e790b4c3-d2e0-4149-9246-7732368e8f30 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:11.012893 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e790b4c3-d2e0-4149-9246-7732368e8f30 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:11.013241 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e790b4c3-d2e0-4149-9246-7732368e8f30 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:11.052904 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-e790b4c3-d2e0-4149-9246-7732368e8f30 None None] MFA Rules not processed for user `c185abe69a6e452dbfac0c057e64d369`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:11.054376 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-e790b4c3-d2e0-4149-9246-7732368e8f30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:11.077035 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-124de92b-75cc-43b0-a0d7-fa6c81574f4d admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:11.143637 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 776/1563] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:11 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:11.145963 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e790b4c3-d2e0-4149-9246-7732368e8f30 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:11.160698 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-124de92b-75cc-43b0-a0d7-fa6c81574f4d admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:11.164114 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e790b4c3-d2e0-4149-9246-7732368e8f30 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:11.165455 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2599f45d-897d-4c61-92b7-6542fef2dc52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:11.166170 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2599f45d-897d-4c61-92b7-6542fef2dc52 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:11.166589 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2599f45d-897d-4c61-92b7-6542fef2dc52 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:11.166967 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2599f45d-897d-4c61-92b7-6542fef2dc52 admin admin] PATH_INFO: `/v3/users/598f0b30f9d34057bb75d6131137ed68` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:11.168325 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2599f45d-897d-4c61-92b7-6542fef2dc52 admin admin] RBAC: Authorizing `identity:delete_user(user_id=598f0b30f9d34057bb75d6131137ed68)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:11.170264 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2599f45d-897d-4c61-92b7-6542fef2dc52 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:11.170714 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b18b1a6a-6fa3-4994-8447-c0d408220e21 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:11.171471 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18b1a6a-6fa3-4994-8447-c0d408220e21 admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:11.171934 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18b1a6a-6fa3-4994-8447-c0d408220e21 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:11.172534 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18b1a6a-6fa3-4994-8447-c0d408220e21 admin admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:11.173303 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b18b1a6a-6fa3-4994-8447-c0d408220e21 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:11.176038 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b18b1a6a-6fa3-4994-8447-c0d408220e21 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:11.241476 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 777/1564] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:01:11 2022] POST /identity/v3/projects => generated 363 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:01:11.242253 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b18b1a6a-6fa3-4994-8447-c0d408220e21 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:11.253287 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-2599f45d-897d-4c61-92b7-6542fef2dc52 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '598f0b30f9d34057bb75d6131137ed68'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:11.271646 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-2599f45d-897d-4c61-92b7-6542fef2dc52 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '598f0b30f9d34057bb75d6131137ed68'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:11.301023 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-2599f45d-897d-4c61-92b7-6542fef2dc52 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '598f0b30f9d34057bb75d6131137ed68'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:11.308948 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:11.316062 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-2599f45d-897d-4c61-92b7-6542fef2dc52 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '598f0b30f9d34057bb75d6131137ed68'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:11.317264 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-2599f45d-897d-4c61-92b7-6542fef2dc52 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '598f0b30f9d34057bb75d6131137ed68'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:11.325973 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 788/1565] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:01:11 2022] DELETE /identity/v3/users/598f0b30f9d34057bb75d6131137ed68 => generated 0 bytes in 249 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:01:11.331403 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aec2add-10a0-431f-b971-a6e8c9507178 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:11.331934 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aec2add-10a0-431f-b971-a6e8c9507178 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:11.332320 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aec2add-10a0-431f-b971-a6e8c9507178 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:11.347316 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b18b1a6a-6fa3-4994-8447-c0d408220e21 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:11.364005 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-3aec2add-10a0-431f-b971-a6e8c9507178 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:11.365362 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-3aec2add-10a0-431f-b971-a6e8c9507178 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:11.400264 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:INSERT=5,keystone:SELECT=85,keystone:DELETE=10 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:11.460470 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a60d88c6-53e9-487a-ae2c-8d8fcd90c20d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:11.462357 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60d88c6-53e9-487a-ae2c-8d8fcd90c20d admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:11.462693 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60d88c6-53e9-487a-ae2c-8d8fcd90c20d admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:11.463002 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60d88c6-53e9-487a-ae2c-8d8fcd90c20d admin admin] PATH_INFO: `/v3/users` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:11.463723 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a60d88c6-53e9-487a-ae2c-8d8fcd90c20d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:11.466167 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a60d88c6-53e9-487a-ae2c-8d8fcd90c20d admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:11.475269 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 789/1566] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:11.502497 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 778/1567] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:01:11 2022] POST /identity/v3/users => generated 288 bytes in 261 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:01:11.506050 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aec2add-10a0-431f-b971-a6e8c9507178 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:11.507555 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-3aec2add-10a0-431f-b971-a6e8c9507178 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:11.513524 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a60d88c6-53e9-487a-ae2c-8d8fcd90c20d admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:11.516845 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-a60d88c6-53e9-487a-ae2c-8d8fcd90c20d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:11.524461 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aec2add-10a0-431f-b971-a6e8c9507178 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:11.530221 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-138317bd-a6cc-435a-bc58-e5cefea4f3c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:11.531361 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-138317bd-a6cc-435a-bc58-e5cefea4f3c2 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:11.531361 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-138317bd-a6cc-435a-bc58-e5cefea4f3c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:11.531603 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-138317bd-a6cc-435a-bc58-e5cefea4f3c2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:11.534642 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-138317bd-a6cc-435a-bc58-e5cefea4f3c2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:11.534827 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-138317bd-a6cc-435a-bc58-e5cefea4f3c2 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:11.549332 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 790/1568] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:01:11 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:01:11.553991 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-138317bd-a6cc-435a-bc58-e5cefea4f3c2 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:11.585817 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-138317bd-a6cc-435a-bc58-e5cefea4f3c2 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:11.595857 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b47cb06-6d35-43ce-8198-e2adf2447629 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:11.596361 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b47cb06-6d35-43ce-8198-e2adf2447629 admin admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:11.596646 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b47cb06-6d35-43ce-8198-e2adf2447629 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:11.596903 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b47cb06-6d35-43ce-8198-e2adf2447629 admin admin] PATH_INFO: `/v3/projects/5847784d5e33418085e3a5be19a91d27/users/b0c1d53f7c81422f83cd5a55efb8dfab/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:11.636718 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b47cb06-6d35-43ce-8198-e2adf2447629 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5847784d5e33418085e3a5be19a91d27, user_id=b0c1d53f7c81422f83cd5a55efb8dfab, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:11.639487 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b47cb06-6d35-43ce-8198-e2adf2447629 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:11.647447 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a60d88c6-53e9-487a-ae2c-8d8fcd90c20d admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:11.657445 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5051b398-a653-4e5c-a266-9ef7a7251110 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:11.657445 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5051b398-a653-4e5c-a266-9ef7a7251110 service neutron] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:11.657445 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5051b398-a653-4e5c-a266-9ef7a7251110 service neutron] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:11.657445 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5051b398-a653-4e5c-a266-9ef7a7251110 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:11.666960 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 791/1569] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:01:11 2022] PUT /identity/v3/projects/5847784d5e33418085e3a5be19a91d27/users/b0c1d53f7c81422f83cd5a55efb8dfab/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:11.671584 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b47cb06-6d35-43ce-8198-e2adf2447629 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:11.683530 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5051b398-a653-4e5c-a266-9ef7a7251110 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:11.686185 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5051b398-a653-4e5c-a266-9ef7a7251110 service neutron] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:11.706458 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 779/1570] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:01:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 193 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:01:11.767904 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b47cb06-6d35-43ce-8198-e2adf2447629 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:11.772940 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0610e6a6-6be0-40e4-9b68-4e21602453f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:11.773792 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0610e6a6-6be0-40e4-9b68-4e21602453f7 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:11.774209 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0610e6a6-6be0-40e4-9b68-4e21602453f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:11.774673 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0610e6a6-6be0-40e4-9b68-4e21602453f7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:11.775703 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0610e6a6-6be0-40e4-9b68-4e21602453f7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:11.778147 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0610e6a6-6be0-40e4-9b68-4e21602453f7 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:11.790643 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 792/1571] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:01:11 2022] GET /identity/v3/roles => generated 1420 bytes in 119 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 20 09:01:11.794507 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5051b398-a653-4e5c-a266-9ef7a7251110 service neutron] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:11.806720 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5051b398-a653-4e5c-a266-9ef7a7251110 service neutron] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:11.812007 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dbc1fbd-d558-4e20-837e-6ebf29b81a6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:11.813331 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dbc1fbd-d558-4e20-837e-6ebf29b81a6b admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:11.813610 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dbc1fbd-d558-4e20-837e-6ebf29b81a6b admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:11.813940 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dbc1fbd-d558-4e20-837e-6ebf29b81a6b admin admin] PATH_INFO: `/v3/projects/5847784d5e33418085e3a5be19a91d27/users/b0c1d53f7c81422f83cd5a55efb8dfab/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:11.815929 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dbc1fbd-d558-4e20-837e-6ebf29b81a6b admin admin] RBAC: Authorizing `identity:create_grant(project_id=5847784d5e33418085e3a5be19a91d27, user_id=b0c1d53f7c81422f83cd5a55efb8dfab, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:11.819165 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dbc1fbd-d558-4e20-837e-6ebf29b81a6b admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:11.833852 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 780/1572] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:01:11 2022] PUT /identity/v3/projects/5847784d5e33418085e3a5be19a91d27/users/b0c1d53f7c81422f83cd5a55efb8dfab/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 40 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:11.845323 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f28ce99a-fec8-4361-8d2f-5d0770e6ea3a None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:11.846376 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f28ce99a-fec8-4361-8d2f-5d0770e6ea3a None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:11.846376 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f28ce99a-fec8-4361-8d2f-5d0770e6ea3a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:11.878710 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-f28ce99a-fec8-4361-8d2f-5d0770e6ea3a None None] MFA Rules not processed for user `b0c1d53f7c81422f83cd5a55efb8dfab`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:11.879930 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-f28ce99a-fec8-4361-8d2f-5d0770e6ea3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:12.012207 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 793/1573] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:11 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:12.027214 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dbc1fbd-d558-4e20-837e-6ebf29b81a6b admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:12.029087 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-8dbc1fbd-d558-4e20-837e-6ebf29b81a6b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:12.112221 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dbc1fbd-d558-4e20-837e-6ebf29b81a6b admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:12.116952 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdfd2b3c-b87d-4e78-a7e2-255850705d2e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:12.118358 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdfd2b3c-b87d-4e78-a7e2-255850705d2e service cinder] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:12.118782 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdfd2b3c-b87d-4e78-a7e2-255850705d2e service cinder] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:12.119057 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdfd2b3c-b87d-4e78-a7e2-255850705d2e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:12.129773 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdfd2b3c-b87d-4e78-a7e2-255850705d2e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:12.131826 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdfd2b3c-b87d-4e78-a7e2-255850705d2e service cinder] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:12.145006 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 781/1574] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:01:12 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:01:12.249098 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f28ce99a-fec8-4361-8d2f-5d0770e6ea3a None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:12.260289 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f28ce99a-fec8-4361-8d2f-5d0770e6ea3a None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:12.267095 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba08b57-57c5-42a8-a9e5-844bb4dc80c0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:12.270646 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba08b57-57c5-42a8-a9e5-844bb4dc80c0 service cinder] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:12.271354 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba08b57-57c5-42a8-a9e5-844bb4dc80c0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:12.271958 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba08b57-57c5-42a8-a9e5-844bb4dc80c0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:12.276998 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-1ba08b57-57c5-42a8-a9e5-844bb4dc80c0 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:12.287591 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:12.370006 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ba08b57-57c5-42a8-a9e5-844bb4dc80c0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:12.371667 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ba08b57-57c5-42a8-a9e5-844bb4dc80c0 service cinder] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:12.384268 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 794/1575] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:01:12 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 136 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:01:12.417891 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdfd2b3c-b87d-4e78-a7e2-255850705d2e service cinder] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:12.418875 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-fdfd2b3c-b87d-4e78-a7e2-255850705d2e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:12.433094 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:12.528325 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdfd2b3c-b87d-4e78-a7e2-255850705d2e service cinder] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:12.541104 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff78a8ae-1005-4d9c-9056-0941470820f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:12.541874 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff78a8ae-1005-4d9c-9056-0941470820f5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:12.542156 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff78a8ae-1005-4d9c-9056-0941470820f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:12.544559 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff78a8ae-1005-4d9c-9056-0941470820f5 admin admin] PATH_INFO: `/v3/projects/7b1ffd8e666743fdbc24a4cde3a37192` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:12.545808 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff78a8ae-1005-4d9c-9056-0941470820f5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7b1ffd8e666743fdbc24a4cde3a37192)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:12.551486 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff78a8ae-1005-4d9c-9056-0941470820f5 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:12.625660 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-ff78a8ae-1005-4d9c-9056-0941470820f5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7b1ffd8e666743fdbc24a4cde3a37192', 'request_id': 'req-ff78a8ae-1005-4d9c-9056-0941470820f5'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:12.638370 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-ff78a8ae-1005-4d9c-9056-0941470820f5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7b1ffd8e666743fdbc24a4cde3a37192', 'request_id': 'req-ff78a8ae-1005-4d9c-9056-0941470820f5'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:12.647624 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-ff78a8ae-1005-4d9c-9056-0941470820f5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7b1ffd8e666743fdbc24a4cde3a37192', 'request_id': 'req-ff78a8ae-1005-4d9c-9056-0941470820f5'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:12.662794 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-ff78a8ae-1005-4d9c-9056-0941470820f5 admin admin] The token cache is being invalidate because project 7b1ffd8e666743fdbc24a4cde3a37192 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81561) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:01:12.663105 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-ff78a8ae-1005-4d9c-9056-0941470820f5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:12.663742 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-ff78a8ae-1005-4d9c-9056-0941470820f5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:12.666514 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 782/1576] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:01:12 2022] DELETE /identity/v3/projects/7b1ffd8e666743fdbc24a4cde3a37192 => generated 0 bytes in 250 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:12.669967 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba08b57-57c5-42a8-a9e5-844bb4dc80c0 service cinder] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:12.681598 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba08b57-57c5-42a8-a9e5-844bb4dc80c0 service cinder] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:12.744938 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f4d58e8-f72a-41bd-b7c6-97adcec3a72d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:12.745588 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4d58e8-f72a-41bd-b7c6-97adcec3a72d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:12.745864 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4d58e8-f72a-41bd-b7c6-97adcec3a72d admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:12.746182 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4d58e8-f72a-41bd-b7c6-97adcec3a72d admin admin] PATH_INFO: `/v3/users/2f12b6c0f67e44fcacd6a3b834a315de` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:12.748186 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f4d58e8-f72a-41bd-b7c6-97adcec3a72d admin admin] RBAC: Authorizing `identity:delete_user(user_id=2f12b6c0f67e44fcacd6a3b834a315de)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:12.749784 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f4d58e8-f72a-41bd-b7c6-97adcec3a72d admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:12.806307 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-9f4d58e8-f72a-41bd-b7c6-97adcec3a72d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2f12b6c0f67e44fcacd6a3b834a315de'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:12.814809 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-9f4d58e8-f72a-41bd-b7c6-97adcec3a72d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2f12b6c0f67e44fcacd6a3b834a315de'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:12.830584 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-9f4d58e8-f72a-41bd-b7c6-97adcec3a72d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2f12b6c0f67e44fcacd6a3b834a315de'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:12.838301 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-9f4d58e8-f72a-41bd-b7c6-97adcec3a72d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2f12b6c0f67e44fcacd6a3b834a315de'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:12.839235 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-9f4d58e8-f72a-41bd-b7c6-97adcec3a72d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2f12b6c0f67e44fcacd6a3b834a315de'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:12.845899 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 795/1577] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:01:12 2022] DELETE /identity/v3/users/2f12b6c0f67e44fcacd6a3b834a315de => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:01:13.133808 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff78a8ae-1005-4d9c-9056-0941470820f5 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:13.135380 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-ff78a8ae-1005-4d9c-9056-0941470820f5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:13.185322 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff78a8ae-1005-4d9c-9056-0941470820f5 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:13.263748 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec2e4fcb-23f1-45a6-aee3-7a6894a4bbb1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:13.264528 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec2e4fcb-23f1-45a6-aee3-7a6894a4bbb1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:13.264830 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec2e4fcb-23f1-45a6-aee3-7a6894a4bbb1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:13.265816 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec2e4fcb-23f1-45a6-aee3-7a6894a4bbb1 admin admin] PATH_INFO: `/v3/projects/2979e4ec6ee54329a498477be56e7a72` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:13.266606 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec2e4fcb-23f1-45a6-aee3-7a6894a4bbb1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2979e4ec6ee54329a498477be56e7a72)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:13.268771 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec2e4fcb-23f1-45a6-aee3-7a6894a4bbb1 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:13.326647 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:13.336629 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-ec2e4fcb-23f1-45a6-aee3-7a6894a4bbb1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2979e4ec6ee54329a498477be56e7a72', 'request_id': 'req-ec2e4fcb-23f1-45a6-aee3-7a6894a4bbb1'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:13.350282 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-ec2e4fcb-23f1-45a6-aee3-7a6894a4bbb1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2979e4ec6ee54329a498477be56e7a72', 'request_id': 'req-ec2e4fcb-23f1-45a6-aee3-7a6894a4bbb1'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:13.373998 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-ec2e4fcb-23f1-45a6-aee3-7a6894a4bbb1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2979e4ec6ee54329a498477be56e7a72', 'request_id': 'req-ec2e4fcb-23f1-45a6-aee3-7a6894a4bbb1'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:13.392582 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-ec2e4fcb-23f1-45a6-aee3-7a6894a4bbb1 admin admin] The token cache is being invalidate because project 2979e4ec6ee54329a498477be56e7a72 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81561) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:01:13.393019 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-ec2e4fcb-23f1-45a6-aee3-7a6894a4bbb1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:13.393843 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-ec2e4fcb-23f1-45a6-aee3-7a6894a4bbb1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:13.398724 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 783/1578] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:01:13 2022] DELETE /identity/v3/projects/2979e4ec6ee54329a498477be56e7a72 => generated 0 bytes in 263 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:13.400616 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed8db5f-931d-42c3-b299-41c90f5736ef None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:13.401111 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed8db5f-931d-42c3-b299-41c90f5736ef None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:13.401204 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed8db5f-931d-42c3-b299-41c90f5736ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:13.423564 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-0ed8db5f-931d-42c3-b299-41c90f5736ef None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:13.425102 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-0ed8db5f-931d-42c3-b299-41c90f5736ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:13.447836 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:13.510919 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 796/1579] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:13.512668 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc4e0f5f-84bb-4365-ad00-e43291b0480a None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:13.512954 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc4e0f5f-84bb-4365-ad00-e43291b0480a None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:13.513229 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc4e0f5f-84bb-4365-ad00-e43291b0480a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:13.535590 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-dc4e0f5f-84bb-4365-ad00-e43291b0480a None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:13.537156 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-dc4e0f5f-84bb-4365-ad00-e43291b0480a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:13.563354 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 784/1580] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 52 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:13.568341 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ed8db5f-931d-42c3-b299-41c90f5736ef None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:13.578874 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ed8db5f-931d-42c3-b299-41c90f5736ef None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:13.584647 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81620bf4-4f2b-44a9-8cda-883d06ee81d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:13.585328 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81620bf4-4f2b-44a9-8cda-883d06ee81d5 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:13.585577 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81620bf4-4f2b-44a9-8cda-883d06ee81d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:13.585868 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81620bf4-4f2b-44a9-8cda-883d06ee81d5 admin admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:13.586591 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81620bf4-4f2b-44a9-8cda-883d06ee81d5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 09:01:13.586887 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81620bf4-4f2b-44a9-8cda-883d06ee81d5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:13.588535 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81620bf4-4f2b-44a9-8cda-883d06ee81d5 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:13.605025 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 797/1581] 10.222.0.36 () {58 vars in 1153 bytes} [Thu Oct 20 09:01:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 09:01:13.606416 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc4e0f5f-84bb-4365-ad00-e43291b0480a None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:13.618274 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc4e0f5f-84bb-4365-ad00-e43291b0480a None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:13.623038 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dffa102-3b0f-4086-b8db-55bf8abd8866 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:13.623643 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dffa102-3b0f-4086-b8db-55bf8abd8866 admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:13.623921 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dffa102-3b0f-4086-b8db-55bf8abd8866 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:13.627099 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dffa102-3b0f-4086-b8db-55bf8abd8866 admin admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:13.627825 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dffa102-3b0f-4086-b8db-55bf8abd8866 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:13.629780 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dffa102-3b0f-4086-b8db-55bf8abd8866 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:13.659851 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 785/1582] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:01:13 2022] POST /identity/v3/projects => generated 383 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:01:13.664371 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81620bf4-4f2b-44a9-8cda-883d06ee81d5 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:13.674404 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81620bf4-4f2b-44a9-8cda-883d06ee81d5 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:13.750152 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb48ca5-99aa-4fee-ad92-f20808c12726 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:13.750807 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb48ca5-99aa-4fee-ad92-f20808c12726 admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:13.751080 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb48ca5-99aa-4fee-ad92-f20808c12726 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:13.751343 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb48ca5-99aa-4fee-ad92-f20808c12726 admin admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:13.752037 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cb48ca5-99aa-4fee-ad92-f20808c12726 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:13.756434 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cb48ca5-99aa-4fee-ad92-f20808c12726 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:13.794519 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 798/1583] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:01:13 2022] POST /identity/v3/users => generated 292 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:01:13.796933 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dffa102-3b0f-4086-b8db-55bf8abd8866 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:13.810850 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dffa102-3b0f-4086-b8db-55bf8abd8866 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:13.827636 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b7cbe56-bd45-4523-9126-76336825640a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:13.828535 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b7cbe56-bd45-4523-9126-76336825640a admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:13.829232 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b7cbe56-bd45-4523-9126-76336825640a admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:13.829883 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b7cbe56-bd45-4523-9126-76336825640a admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:13.833876 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b7cbe56-bd45-4523-9126-76336825640a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:13.836144 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b7cbe56-bd45-4523-9126-76336825640a admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:13.852462 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 786/1584] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:01:13 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:01:13.853629 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb48ca5-99aa-4fee-ad92-f20808c12726 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:13.865484 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb48ca5-99aa-4fee-ad92-f20808c12726 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:13.872538 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3e73a1b-a55e-4cfa-ae86-4ed19e885be1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:13.873227 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e73a1b-a55e-4cfa-ae86-4ed19e885be1 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:13.873831 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e73a1b-a55e-4cfa-ae86-4ed19e885be1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:13.874200 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e73a1b-a55e-4cfa-ae86-4ed19e885be1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:13.874976 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3e73a1b-a55e-4cfa-ae86-4ed19e885be1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:13.877410 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3e73a1b-a55e-4cfa-ae86-4ed19e885be1 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:13.901719 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 799/1585] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:01:13 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:01:13.902800 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b7cbe56-bd45-4523-9126-76336825640a admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:13.920983 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b7cbe56-bd45-4523-9126-76336825640a admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:13.926806 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef9e3009-4986-42cd-9760-e3403333881f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:13.927420 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9e3009-4986-42cd-9760-e3403333881f admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:13.927699 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9e3009-4986-42cd-9760-e3403333881f admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:13.927975 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9e3009-4986-42cd-9760-e3403333881f admin admin] PATH_INFO: `/v3/projects/192b294d0db04fbcaf3b125da598ea2b/users/de576ca2e5cc462c910eec4e435f7ac7/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:13.943695 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3e73a1b-a55e-4cfa-ae86-4ed19e885be1 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:13.945888 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-f3e73a1b-a55e-4cfa-ae86-4ed19e885be1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:13.961290 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef9e3009-4986-42cd-9760-e3403333881f admin admin] RBAC: Authorizing `identity:create_grant(project_id=192b294d0db04fbcaf3b125da598ea2b, user_id=de576ca2e5cc462c910eec4e435f7ac7, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:13.963993 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef9e3009-4986-42cd-9760-e3403333881f admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:13.971930 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3e73a1b-a55e-4cfa-ae86-4ed19e885be1 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:13.979120 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b4b89d8-3ea2-4f5a-bbde-7b0dab9d283e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:13.979775 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4b89d8-3ea2-4f5a-bbde-7b0dab9d283e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:13.980055 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4b89d8-3ea2-4f5a-bbde-7b0dab9d283e admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:13.980374 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4b89d8-3ea2-4f5a-bbde-7b0dab9d283e admin admin] PATH_INFO: `/v3/users/874a9c0c0ff7445b9465e67847c3b4c0` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:13.984271 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b4b89d8-3ea2-4f5a-bbde-7b0dab9d283e admin admin] RBAC: Authorizing `identity:delete_user(user_id=874a9c0c0ff7445b9465e67847c3b4c0)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:13.987860 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b4b89d8-3ea2-4f5a-bbde-7b0dab9d283e admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:13.992894 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 787/1586] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:01:13 2022] PUT /identity/v3/projects/192b294d0db04fbcaf3b125da598ea2b/users/de576ca2e5cc462c910eec4e435f7ac7/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 90 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:14.003822 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c31aa5f8-1b44-475c-a7f0-9f3a3e601f8d None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:14.004320 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c31aa5f8-1b44-475c-a7f0-9f3a3e601f8d None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:14.004644 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c31aa5f8-1b44-475c-a7f0-9f3a3e601f8d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:14.090574 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-c31aa5f8-1b44-475c-a7f0-9f3a3e601f8d None None] MFA Rules not processed for user `de576ca2e5cc462c910eec4e435f7ac7`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:14.091712 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-c31aa5f8-1b44-475c-a7f0-9f3a3e601f8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:14.108898 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-3b4b89d8-3ea2-4f5a-bbde-7b0dab9d283e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '874a9c0c0ff7445b9465e67847c3b4c0'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:14.121363 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-3b4b89d8-3ea2-4f5a-bbde-7b0dab9d283e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '874a9c0c0ff7445b9465e67847c3b4c0'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:14.141519 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-3b4b89d8-3ea2-4f5a-bbde-7b0dab9d283e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '874a9c0c0ff7445b9465e67847c3b4c0'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:14.151086 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-3b4b89d8-3ea2-4f5a-bbde-7b0dab9d283e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '874a9c0c0ff7445b9465e67847c3b4c0'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:14.154520 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-3b4b89d8-3ea2-4f5a-bbde-7b0dab9d283e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '874a9c0c0ff7445b9465e67847c3b4c0'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:14.166983 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 800/1587] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:01:13 2022] DELETE /identity/v3/users/874a9c0c0ff7445b9465e67847c3b4c0 => generated 0 bytes in 224 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:14.171554 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25f28464-afbc-40e7-b4b0-d7e26337e6a3 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:14.172148 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25f28464-afbc-40e7-b4b0-d7e26337e6a3 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:14.172574 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25f28464-afbc-40e7-b4b0-d7e26337e6a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:14.188692 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=86,keystone:INSERT=4,keystone:DELETE=10 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:14.197426 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-25f28464-afbc-40e7-b4b0-d7e26337e6a3 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:14.199535 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-25f28464-afbc-40e7-b4b0-d7e26337e6a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:14.207666 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 788/1588] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:14 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 20 09:01:14.213015 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c31aa5f8-1b44-475c-a7f0-9f3a3e601f8d None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:14.217081 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-c31aa5f8-1b44-475c-a7f0-9f3a3e601f8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:14.314961 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 801/1589] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:14.336440 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=93,keystone:DELETE=4,keystone:INSERT=3 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:14.343890 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c31aa5f8-1b44-475c-a7f0-9f3a3e601f8d None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:14.346086 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25f28464-afbc-40e7-b4b0-d7e26337e6a3 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:14.347825 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-25f28464-afbc-40e7-b4b0-d7e26337e6a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:14.352032 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebdd9d59-60fa-4931-bac2-735861e5efeb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:14.356638 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebdd9d59-60fa-4931-bac2-735861e5efeb admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:14.357047 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebdd9d59-60fa-4931-bac2-735861e5efeb admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:14.357601 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebdd9d59-60fa-4931-bac2-735861e5efeb admin admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:14.371874 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebdd9d59-60fa-4931-bac2-735861e5efeb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:14.382261 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebdd9d59-60fa-4931-bac2-735861e5efeb admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:14.486457 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 789/1590] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:01:14 2022] POST /identity/v3/projects => generated 385 bytes in 272 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:01:14.487513 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebdd9d59-60fa-4931-bac2-735861e5efeb admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:14.520015 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25f28464-afbc-40e7-b4b0-d7e26337e6a3 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:14.657428 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2ed084e-6845-4528-a734-51f057207277 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:14.658074 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ed084e-6845-4528-a734-51f057207277 service neutron] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:14.658346 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ed084e-6845-4528-a734-51f057207277 service neutron] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:14.658632 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ed084e-6845-4528-a734-51f057207277 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:14.662399 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebdd9d59-60fa-4931-bac2-735861e5efeb admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:14.682047 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db527498-e6e4-4998-8bbb-23104a89ad89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:14.683160 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db527498-e6e4-4998-8bbb-23104a89ad89 admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:14.683160 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db527498-e6e4-4998-8bbb-23104a89ad89 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:14.683331 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db527498-e6e4-4998-8bbb-23104a89ad89 admin admin] PATH_INFO: `/v3/users` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:14.683965 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db527498-e6e4-4998-8bbb-23104a89ad89 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:14.698108 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db527498-e6e4-4998-8bbb-23104a89ad89 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:14.707364 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2ed084e-6845-4528-a734-51f057207277 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:14.711256 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2ed084e-6845-4528-a734-51f057207277 service neutron] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:14.760559 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 802/1591] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:01:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 415 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:01:14.780556 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 790/1592] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:01:14 2022] POST /identity/v3/users => generated 293 bytes in 290 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 20 09:01:14.781950 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2ed084e-6845-4528-a734-51f057207277 service neutron] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:14.792685 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2ed084e-6845-4528-a734-51f057207277 service neutron] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:14.797908 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2579bac6-ddf0-4331-acf2-99e518c7ae5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:14.798842 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2579bac6-ddf0-4331-acf2-99e518c7ae5b admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:14.799346 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2579bac6-ddf0-4331-acf2-99e518c7ae5b admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:14.799845 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2579bac6-ddf0-4331-acf2-99e518c7ae5b admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:14.801407 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2579bac6-ddf0-4331-acf2-99e518c7ae5b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:14.803506 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2579bac6-ddf0-4331-acf2-99e518c7ae5b admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:14.818604 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 803/1593] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:01:14 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:01:14.824492 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db527498-e6e4-4998-8bbb-23104a89ad89 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:14.836764 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db527498-e6e4-4998-8bbb-23104a89ad89 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:14.845720 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fe0fc0d-90dd-404a-adc9-eea0aacad015 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:14.847735 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe0fc0d-90dd-404a-adc9-eea0aacad015 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:14.848292 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe0fc0d-90dd-404a-adc9-eea0aacad015 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:14.848767 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe0fc0d-90dd-404a-adc9-eea0aacad015 admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:14.850246 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fe0fc0d-90dd-404a-adc9-eea0aacad015 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:14.853070 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fe0fc0d-90dd-404a-adc9-eea0aacad015 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:14.871533 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 791/1594] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:01:14 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:01:14.875773 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2579bac6-ddf0-4331-acf2-99e518c7ae5b admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:14.888543 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2579bac6-ddf0-4331-acf2-99e518c7ae5b admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:14.897667 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3fc286a-d64f-41c3-94f8-3041020639cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:14.898768 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3fc286a-d64f-41c3-94f8-3041020639cf admin admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:14.899277 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3fc286a-d64f-41c3-94f8-3041020639cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:14.899960 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3fc286a-d64f-41c3-94f8-3041020639cf admin admin] PATH_INFO: `/v3/projects/1776e1f445f548b7a54decdabe1c469d/users/fbe1f133cdc048ac804b402a7164ccfa/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:14.930727 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3fc286a-d64f-41c3-94f8-3041020639cf admin admin] RBAC: Authorizing `identity:create_grant(project_id=1776e1f445f548b7a54decdabe1c469d, user_id=fbe1f133cdc048ac804b402a7164ccfa, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:14.935594 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3fc286a-d64f-41c3-94f8-3041020639cf admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:14.956214 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 804/1595] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:01:14 2022] PUT /identity/v3/projects/1776e1f445f548b7a54decdabe1c469d/users/fbe1f133cdc048ac804b402a7164ccfa/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 81 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:14.968005 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a4b1d8-e5e3-40e6-baca-5d279fe5f6a0 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:14.968362 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a4b1d8-e5e3-40e6-baca-5d279fe5f6a0 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:14.968709 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a4b1d8-e5e3-40e6-baca-5d279fe5f6a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:15.008103 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-08a4b1d8-e5e3-40e6-baca-5d279fe5f6a0 None None] MFA Rules not processed for user `fbe1f133cdc048ac804b402a7164ccfa`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:15.009406 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-08a4b1d8-e5e3-40e6-baca-5d279fe5f6a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:15.079788 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 792/1596] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:14 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:15.084383 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3fc286a-d64f-41c3-94f8-3041020639cf admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:15.126270 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:15.172265 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3fc286a-d64f-41c3-94f8-3041020639cf admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:15.190657 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1355e661-c2ab-407e-984a-0b5eadf85e54 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:15.190657 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1355e661-c2ab-407e-984a-0b5eadf85e54 admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:15.190657 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1355e661-c2ab-407e-984a-0b5eadf85e54 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:15.190657 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1355e661-c2ab-407e-984a-0b5eadf85e54 admin admin] PATH_INFO: `/v3/projects` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:15.191178 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1355e661-c2ab-407e-984a-0b5eadf85e54 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:15.191178 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1355e661-c2ab-407e-984a-0b5eadf85e54 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:15.210514 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 805/1597] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:01:15 2022] POST /identity/v3/projects => generated 385 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:01:15.214165 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08a4b1d8-e5e3-40e6-baca-5d279fe5f6a0 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:15.226111 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08a4b1d8-e5e3-40e6-baca-5d279fe5f6a0 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:15.247395 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1355e661-c2ab-407e-984a-0b5eadf85e54 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:15.250028 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-1355e661-c2ab-407e-984a-0b5eadf85e54 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:15.288208 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:15.303620 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e9ad52d-bb52-4818-b9c9-48c390af1b47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:15.304233 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9ad52d-bb52-4818-b9c9-48c390af1b47 admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:15.304540 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9ad52d-bb52-4818-b9c9-48c390af1b47 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:15.304811 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9ad52d-bb52-4818-b9c9-48c390af1b47 admin admin] PATH_INFO: `/v3/users` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:15.305471 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e9ad52d-bb52-4818-b9c9-48c390af1b47 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:15.307690 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e9ad52d-bb52-4818-b9c9-48c390af1b47 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:15.319573 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1355e661-c2ab-407e-984a-0b5eadf85e54 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:15.325130 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d64c7660-d7ce-4436-bbba-12b53d47d93a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:15.325918 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64c7660-d7ce-4436-bbba-12b53d47d93a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:15.326198 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64c7660-d7ce-4436-bbba-12b53d47d93a admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:15.326486 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64c7660-d7ce-4436-bbba-12b53d47d93a admin admin] PATH_INFO: `/v3/projects/379e05649b904064be9a1055ba59b895` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:15.327796 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d64c7660-d7ce-4436-bbba-12b53d47d93a admin admin] RBAC: Authorizing `identity:delete_project(project_id=379e05649b904064be9a1055ba59b895)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:15.329724 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d64c7660-d7ce-4436-bbba-12b53d47d93a admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:15.333006 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 793/1598] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:01:15 2022] POST /identity/v3/users => generated 299 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:01:15.334028 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e9ad52d-bb52-4818-b9c9-48c390af1b47 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:15.347002 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e9ad52d-bb52-4818-b9c9-48c390af1b47 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:15.352447 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed009c3a-83c4-4ee1-a520-4d1b7da92c1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:15.353002 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed009c3a-83c4-4ee1-a520-4d1b7da92c1c admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:15.353280 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed009c3a-83c4-4ee1-a520-4d1b7da92c1c admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:15.353547 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed009c3a-83c4-4ee1-a520-4d1b7da92c1c admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:15.354186 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed009c3a-83c4-4ee1-a520-4d1b7da92c1c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:15.355833 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed009c3a-83c4-4ee1-a520-4d1b7da92c1c admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:15.369759 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 794/1599] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:01:15 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:01:15.371503 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed009c3a-83c4-4ee1-a520-4d1b7da92c1c admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:15.383917 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed009c3a-83c4-4ee1-a520-4d1b7da92c1c admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:15.395860 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-d64c7660-d7ce-4436-bbba-12b53d47d93a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '379e05649b904064be9a1055ba59b895', 'request_id': 'req-d64c7660-d7ce-4436-bbba-12b53d47d93a'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:15.405999 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-d64c7660-d7ce-4436-bbba-12b53d47d93a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '379e05649b904064be9a1055ba59b895', 'request_id': 'req-d64c7660-d7ce-4436-bbba-12b53d47d93a'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:15.414766 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-d64c7660-d7ce-4436-bbba-12b53d47d93a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '379e05649b904064be9a1055ba59b895', 'request_id': 'req-d64c7660-d7ce-4436-bbba-12b53d47d93a'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:15.445446 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-d64c7660-d7ce-4436-bbba-12b53d47d93a admin admin] The token cache is being invalidate because project 379e05649b904064be9a1055ba59b895 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81562) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:01:15.445739 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-d64c7660-d7ce-4436-bbba-12b53d47d93a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:15.450893 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-d64c7660-d7ce-4436-bbba-12b53d47d93a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:15.453368 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 806/1600] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:01:15 2022] DELETE /identity/v3/projects/379e05649b904064be9a1055ba59b895 => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:15.460428 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d64c7660-d7ce-4436-bbba-12b53d47d93a admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:15.462500 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-d64c7660-d7ce-4436-bbba-12b53d47d93a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:15.489600 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2fbdb83-bafd-41b0-8a08-0c8680c0c1fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:15.491352 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2fbdb83-bafd-41b0-8a08-0c8680c0c1fe admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:15.491641 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2fbdb83-bafd-41b0-8a08-0c8680c0c1fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:15.491954 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2fbdb83-bafd-41b0-8a08-0c8680c0c1fe admin admin] PATH_INFO: `/v3/projects/09b8131fb64e44599bc99a9fbe6f22d1/users/f91335f09b5141d9aa84abb0bacb2215/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:15.518100 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2fbdb83-bafd-41b0-8a08-0c8680c0c1fe admin admin] RBAC: Authorizing `identity:create_grant(project_id=09b8131fb64e44599bc99a9fbe6f22d1, user_id=f91335f09b5141d9aa84abb0bacb2215, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:15.521529 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2fbdb83-bafd-41b0-8a08-0c8680c0c1fe admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:15.540759 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:15.547602 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 795/1601] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:01:15 2022] PUT /identity/v3/projects/09b8131fb64e44599bc99a9fbe6f22d1/users/f91335f09b5141d9aa84abb0bacb2215/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:15.551879 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2fbdb83-bafd-41b0-8a08-0c8680c0c1fe admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:15.553956 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-b2fbdb83-bafd-41b0-8a08-0c8680c0c1fe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:15.569899 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d64c7660-d7ce-4436-bbba-12b53d47d93a admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:15.666833 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c471f2c5-c859-4f86-a2fb-742a1d2ad7d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:15.668846 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c471f2c5-c859-4f86-a2fb-742a1d2ad7d2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:15.669131 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c471f2c5-c859-4f86-a2fb-742a1d2ad7d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:15.669404 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c471f2c5-c859-4f86-a2fb-742a1d2ad7d2 admin admin] PATH_INFO: `/v3/users/10e0cfe4fbfc417c9ec4a7038e5efe5b` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:15.671141 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c471f2c5-c859-4f86-a2fb-742a1d2ad7d2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=10e0cfe4fbfc417c9ec4a7038e5efe5b)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:15.673102 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c471f2c5-c859-4f86-a2fb-742a1d2ad7d2 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:15.676990 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2fbdb83-bafd-41b0-8a08-0c8680c0c1fe admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:15.684108 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfd62a96-f719-4ee0-acd1-a95d0c76fb5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:15.685569 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfd62a96-f719-4ee0-acd1-a95d0c76fb5c admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:15.686677 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfd62a96-f719-4ee0-acd1-a95d0c76fb5c admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:15.690968 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfd62a96-f719-4ee0-acd1-a95d0c76fb5c admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:15.691779 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfd62a96-f719-4ee0-acd1-a95d0c76fb5c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:15.693442 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfd62a96-f719-4ee0-acd1-a95d0c76fb5c admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:15.705416 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 796/1602] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:01:15 2022] GET /identity/v3/roles => generated 1420 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:01:15.709631 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfd62a96-f719-4ee0-acd1-a95d0c76fb5c admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:15.746515 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfd62a96-f719-4ee0-acd1-a95d0c76fb5c admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:15.765344 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6e965ac-4535-474a-8e44-959ad672d7bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:15.766316 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e965ac-4535-474a-8e44-959ad672d7bb admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:15.766668 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e965ac-4535-474a-8e44-959ad672d7bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:15.767861 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e965ac-4535-474a-8e44-959ad672d7bb admin admin] PATH_INFO: `/v3/projects/09b8131fb64e44599bc99a9fbe6f22d1/users/f91335f09b5141d9aa84abb0bacb2215/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:15.770810 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6e965ac-4535-474a-8e44-959ad672d7bb admin admin] RBAC: Authorizing `identity:create_grant(project_id=09b8131fb64e44599bc99a9fbe6f22d1, user_id=f91335f09b5141d9aa84abb0bacb2215, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:15.782279 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-c471f2c5-c859-4f86-a2fb-742a1d2ad7d2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '10e0cfe4fbfc417c9ec4a7038e5efe5b'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:15.784774 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6e965ac-4535-474a-8e44-959ad672d7bb admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:15.814028 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-c471f2c5-c859-4f86-a2fb-742a1d2ad7d2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '10e0cfe4fbfc417c9ec4a7038e5efe5b'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:15.844307 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 797/1603] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:01:15 2022] PUT /identity/v3/projects/09b8131fb64e44599bc99a9fbe6f22d1/users/f91335f09b5141d9aa84abb0bacb2215/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:01:15.846125 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71fd37be-6911-46d0-bd01-cec0f9579013 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:15.846409 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71fd37be-6911-46d0-bd01-cec0f9579013 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:15.849206 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71fd37be-6911-46d0-bd01-cec0f9579013 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:15.867341 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-c471f2c5-c859-4f86-a2fb-742a1d2ad7d2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '10e0cfe4fbfc417c9ec4a7038e5efe5b'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:15.895549 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-c471f2c5-c859-4f86-a2fb-742a1d2ad7d2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '10e0cfe4fbfc417c9ec4a7038e5efe5b'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:15.896747 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-c471f2c5-c859-4f86-a2fb-742a1d2ad7d2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '10e0cfe4fbfc417c9ec4a7038e5efe5b'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:15.917884 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 807/1604] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:01:15 2022] DELETE /identity/v3/users/10e0cfe4fbfc417c9ec4a7038e5efe5b => generated 0 bytes in 455 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:01:15.919471 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-71fd37be-6911-46d0-bd01-cec0f9579013 None None] MFA Rules not processed for user `f91335f09b5141d9aa84abb0bacb2215`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:15.921079 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-71fd37be-6911-46d0-bd01-cec0f9579013 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:15.962891 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:16.109143 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 798/1605] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:15 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 264 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:16.121703 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c471f2c5-c859-4f86-a2fb-742a1d2ad7d2 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:16.123203 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-c471f2c5-c859-4f86-a2fb-742a1d2ad7d2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:16.254449 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c471f2c5-c859-4f86-a2fb-742a1d2ad7d2 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:16.265188 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe38775b-dbc2-4277-94da-84a7c1ba6d3c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:16.265820 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe38775b-dbc2-4277-94da-84a7c1ba6d3c service cinder] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:16.266102 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe38775b-dbc2-4277-94da-84a7c1ba6d3c service cinder] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:16.266380 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe38775b-dbc2-4277-94da-84a7c1ba6d3c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:16.269142 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-fe38775b-dbc2-4277-94da-84a7c1ba6d3c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:16.345124 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:16.461840 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe38775b-dbc2-4277-94da-84a7c1ba6d3c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:16.463482 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe38775b-dbc2-4277-94da-84a7c1ba6d3c service cinder] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:16.488598 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d13a546-c838-4cfd-8868-d12937d1c58a None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:16.489118 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d13a546-c838-4cfd-8868-d12937d1c58a None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:16.489385 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d13a546-c838-4cfd-8868-d12937d1c58a None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:16.492533 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 799/1606] 10.222.0.36 () {54 vars in 921 bytes} [Thu Oct 20 09:01:16 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 09:01:16.499893 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 808/1607] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:01:16 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 378 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:01:16.508576 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e65127-8607-452e-9da9-213134b54121 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:16.508857 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e65127-8607-452e-9da9-213134b54121 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:16.509126 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e65127-8607-452e-9da9-213134b54121 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:16.554978 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-c1e65127-8607-452e-9da9-213134b54121 None None] MFA Rules not processed for user `a94e868115ec49918eda10724263930c`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:16.556224 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-c1e65127-8607-452e-9da9-213134b54121 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:16.788808 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 800/1608] 10.222.0.36 () {58 vars in 1018 bytes} [Thu Oct 20 09:01:16 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 282 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:16.811857 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe38775b-dbc2-4277-94da-84a7c1ba6d3c service cinder] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:16.814030 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-fe38775b-dbc2-4277-94da-84a7c1ba6d3c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:16.851028 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e65127-8607-452e-9da9-213134b54121 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:16.852865 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-c1e65127-8607-452e-9da9-213134b54121 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:16.908220 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e65127-8607-452e-9da9-213134b54121 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:16.925502 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e35d40c-0556-4d57-ad3a-e854986d1ff8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a94e868115ec49918eda10724263930c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:16.928888 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e35d40c-0556-4d57-ad3a-e854986d1ff8 service nova] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:16.931067 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e35d40c-0556-4d57-ad3a-e854986d1ff8 service nova] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:16.931903 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e35d40c-0556-4d57-ad3a-e854986d1ff8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:16.957866 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e35d40c-0556-4d57-ad3a-e854986d1ff8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:16.962169 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e35d40c-0556-4d57-ad3a-e854986d1ff8 service nova] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:16.976920 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe38775b-dbc2-4277-94da-84a7c1ba6d3c service cinder] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:16.982919 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-149273bc-c484-436a-a1a8-f5e130d05f02 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:16.983767 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-149273bc-c484-436a-a1a8-f5e130d05f02 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:16.984376 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-149273bc-c484-436a-a1a8-f5e130d05f02 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:16.984894 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-149273bc-c484-436a-a1a8-f5e130d05f02 admin admin] PATH_INFO: `/v3/projects/a7907859f9e7402bbdd9921a221ed75c` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:16.986362 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-149273bc-c484-436a-a1a8-f5e130d05f02 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a7907859f9e7402bbdd9921a221ed75c)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:16.989056 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-149273bc-c484-436a-a1a8-f5e130d05f02 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:16.998249 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 801/1609] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:01:16 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 147 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:01:17.087486 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-149273bc-c484-436a-a1a8-f5e130d05f02 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a7907859f9e7402bbdd9921a221ed75c', 'request_id': 'req-149273bc-c484-436a-a1a8-f5e130d05f02'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:17.096725 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-149273bc-c484-436a-a1a8-f5e130d05f02 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a7907859f9e7402bbdd9921a221ed75c', 'request_id': 'req-149273bc-c484-436a-a1a8-f5e130d05f02'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:17.103935 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-149273bc-c484-436a-a1a8-f5e130d05f02 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a7907859f9e7402bbdd9921a221ed75c', 'request_id': 'req-149273bc-c484-436a-a1a8-f5e130d05f02'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:17.123261 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-149273bc-c484-436a-a1a8-f5e130d05f02 admin admin] The token cache is being invalidate because project a7907859f9e7402bbdd9921a221ed75c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81562) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:01:17.123727 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-149273bc-c484-436a-a1a8-f5e130d05f02 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:17.126124 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-149273bc-c484-436a-a1a8-f5e130d05f02 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:17.130265 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 809/1610] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:01:16 2022] DELETE /identity/v3/projects/a7907859f9e7402bbdd9921a221ed75c => generated 0 bytes in 319 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:17.152224 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-026b19ed-ce59-4a09-a68a-b0c3d04075bc None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:17.152535 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-026b19ed-ce59-4a09-a68a-b0c3d04075bc None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:17.152781 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-026b19ed-ce59-4a09-a68a-b0c3d04075bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:17.242661 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-026b19ed-ce59-4a09-a68a-b0c3d04075bc None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:17.251398 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-026b19ed-ce59-4a09-a68a-b0c3d04075bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:17.369916 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 802/1611] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 219 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:17.377436 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22267361-3d82-4903-aadd-e760db27fff1 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:17.377650 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22267361-3d82-4903-aadd-e760db27fff1 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:17.377957 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22267361-3d82-4903-aadd-e760db27fff1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:17.408522 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-22267361-3d82-4903-aadd-e760db27fff1 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:17.409920 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-22267361-3d82-4903-aadd-e760db27fff1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:17.434362 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 810/1612] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 58 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:17.434850 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-026b19ed-ce59-4a09-a68a-b0c3d04075bc None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:17.446002 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-026b19ed-ce59-4a09-a68a-b0c3d04075bc None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:17.451344 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66cc98e3-6c53-4e4b-bffa-f94e0847a176 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:17.451954 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66cc98e3-6c53-4e4b-bffa-f94e0847a176 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:17.452227 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66cc98e3-6c53-4e4b-bffa-f94e0847a176 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:17.452494 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66cc98e3-6c53-4e4b-bffa-f94e0847a176 admin admin] PATH_INFO: `/v3/domains` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:17.453234 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66cc98e3-6c53-4e4b-bffa-f94e0847a176 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 09:01:17.453542 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66cc98e3-6c53-4e4b-bffa-f94e0847a176 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:17.455439 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66cc98e3-6c53-4e4b-bffa-f94e0847a176 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:17.468222 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 803/1613] 10.222.0.36 () {58 vars in 1153 bytes} [Thu Oct 20 09:01:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 09:01:17.472727 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22267361-3d82-4903-aadd-e760db27fff1 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:17.500440 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22267361-3d82-4903-aadd-e760db27fff1 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:17.509836 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c152e3d-b35d-45d0-a60f-091ab4316c14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:17.512211 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c152e3d-b35d-45d0-a60f-091ab4316c14 admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:17.512755 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c152e3d-b35d-45d0-a60f-091ab4316c14 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:17.513174 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c152e3d-b35d-45d0-a60f-091ab4316c14 admin admin] PATH_INFO: `/v3/projects` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:17.515103 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c152e3d-b35d-45d0-a60f-091ab4316c14 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:17.519329 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c152e3d-b35d-45d0-a60f-091ab4316c14 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:17.568818 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 811/1614] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:01:17 2022] POST /identity/v3/projects => generated 369 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:01:17.572728 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66cc98e3-6c53-4e4b-bffa-f94e0847a176 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:17.601805 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66cc98e3-6c53-4e4b-bffa-f94e0847a176 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:17.653626 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:17.709629 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64979669-87cf-41da-b78f-4ab39db65e7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:17.711629 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64979669-87cf-41da-b78f-4ab39db65e7a admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:17.712052 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64979669-87cf-41da-b78f-4ab39db65e7a admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:17.712369 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64979669-87cf-41da-b78f-4ab39db65e7a admin admin] PATH_INFO: `/v3/users` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:17.713294 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64979669-87cf-41da-b78f-4ab39db65e7a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:17.716659 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64979669-87cf-41da-b78f-4ab39db65e7a admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:17.747229 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 804/1615] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:01:17 2022] POST /identity/v3/users => generated 285 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:01:17.749017 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c152e3d-b35d-45d0-a60f-091ab4316c14 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:17.762416 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c152e3d-b35d-45d0-a60f-091ab4316c14 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:17.767811 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be07a51a-296d-4124-9adc-c44337fe6a76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:17.768521 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be07a51a-296d-4124-9adc-c44337fe6a76 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:17.768884 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be07a51a-296d-4124-9adc-c44337fe6a76 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:17.769519 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be07a51a-296d-4124-9adc-c44337fe6a76 admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:17.769930 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be07a51a-296d-4124-9adc-c44337fe6a76 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:17.771780 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be07a51a-296d-4124-9adc-c44337fe6a76 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:17.788227 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 812/1616] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:01:17 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:01:17.789711 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64979669-87cf-41da-b78f-4ab39db65e7a admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:17.800878 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64979669-87cf-41da-b78f-4ab39db65e7a admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:17.805569 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b5f57d-e6e6-45bf-9544-81d71c03e796 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:17.806304 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b5f57d-e6e6-45bf-9544-81d71c03e796 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:17.806752 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b5f57d-e6e6-45bf-9544-81d71c03e796 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:17.807027 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b5f57d-e6e6-45bf-9544-81d71c03e796 admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:17.807667 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97b5f57d-e6e6-45bf-9544-81d71c03e796 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:17.809288 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97b5f57d-e6e6-45bf-9544-81d71c03e796 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:17.820258 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 805/1617] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:01:17 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 20 09:01:17.825250 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be07a51a-296d-4124-9adc-c44337fe6a76 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:17.837891 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be07a51a-296d-4124-9adc-c44337fe6a76 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:17.843282 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c78b0b74-8621-4e6a-aa0f-4de341f0e514 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:17.843947 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c78b0b74-8621-4e6a-aa0f-4de341f0e514 admin admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:17.844223 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c78b0b74-8621-4e6a-aa0f-4de341f0e514 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:17.844493 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c78b0b74-8621-4e6a-aa0f-4de341f0e514 admin admin] PATH_INFO: `/v3/projects/26b9b0f59a9345418c8c423d277cfbae/users/c6407ae2e0fb4b2f8d729f804c672302/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:17.865416 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c78b0b74-8621-4e6a-aa0f-4de341f0e514 admin admin] RBAC: Authorizing `identity:create_grant(project_id=26b9b0f59a9345418c8c423d277cfbae, user_id=c6407ae2e0fb4b2f8d729f804c672302, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:17.866141 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:17.869872 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c78b0b74-8621-4e6a-aa0f-4de341f0e514 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:17.888019 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 813/1618] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:01:17 2022] PUT /identity/v3/projects/26b9b0f59a9345418c8c423d277cfbae/users/c6407ae2e0fb4b2f8d729f804c672302/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:17.898893 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcc0bdb9-aa88-4ac6-a152-25b51dffc820 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:17.899182 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcc0bdb9-aa88-4ac6-a152-25b51dffc820 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:17.899779 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcc0bdb9-aa88-4ac6-a152-25b51dffc820 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:17.944557 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-dcc0bdb9-aa88-4ac6-a152-25b51dffc820 None None] MFA Rules not processed for user `c6407ae2e0fb4b2f8d729f804c672302`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:17.946360 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-dcc0bdb9-aa88-4ac6-a152-25b51dffc820 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:18.063572 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 806/1619] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:17 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:18.063969 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c78b0b74-8621-4e6a-aa0f-4de341f0e514 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:18.147702 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c78b0b74-8621-4e6a-aa0f-4de341f0e514 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:18.156123 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87e1e927-82f0-4d96-8b6e-90673ff6434a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:18.156735 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e1e927-82f0-4d96-8b6e-90673ff6434a admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:18.157008 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e1e927-82f0-4d96-8b6e-90673ff6434a admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:18.157271 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e1e927-82f0-4d96-8b6e-90673ff6434a admin admin] PATH_INFO: `/v3/projects` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:18.157904 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87e1e927-82f0-4d96-8b6e-90673ff6434a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:18.159850 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87e1e927-82f0-4d96-8b6e-90673ff6434a admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:18.187100 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 814/1620] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:01:18 2022] POST /identity/v3/projects => generated 367 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 20 09:01:18.191269 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcc0bdb9-aa88-4ac6-a152-25b51dffc820 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:18.202564 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcc0bdb9-aa88-4ac6-a152-25b51dffc820 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:18.263369 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d769a87-5ea3-4b20-8edd-a2a14797e70d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:18.266737 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d769a87-5ea3-4b20-8edd-a2a14797e70d admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:18.266737 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d769a87-5ea3-4b20-8edd-a2a14797e70d admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:18.266737 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d769a87-5ea3-4b20-8edd-a2a14797e70d admin admin] PATH_INFO: `/v3/users` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:18.266737 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d769a87-5ea3-4b20-8edd-a2a14797e70d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:18.267381 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d769a87-5ea3-4b20-8edd-a2a14797e70d admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:18.293119 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 807/1621] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:01:18 2022] POST /identity/v3/users => generated 290 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:01:18.297732 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87e1e927-82f0-4d96-8b6e-90673ff6434a admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:18.321930 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87e1e927-82f0-4d96-8b6e-90673ff6434a admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:18.329422 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-256345a7-4146-4367-94d4-d2d54913c0b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:18.330150 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-256345a7-4146-4367-94d4-d2d54913c0b7 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:18.330557 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-256345a7-4146-4367-94d4-d2d54913c0b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:18.331033 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-256345a7-4146-4367-94d4-d2d54913c0b7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:18.332140 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-256345a7-4146-4367-94d4-d2d54913c0b7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:18.334292 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-256345a7-4146-4367-94d4-d2d54913c0b7 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:18.350131 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 815/1622] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:01:18 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:01:18.357335 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d769a87-5ea3-4b20-8edd-a2a14797e70d admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:18.372174 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d769a87-5ea3-4b20-8edd-a2a14797e70d admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:18.380799 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57fef3a4-0138-4235-a963-df214ffcbd1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:18.381477 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57fef3a4-0138-4235-a963-df214ffcbd1d admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:18.381754 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57fef3a4-0138-4235-a963-df214ffcbd1d admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:18.382045 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57fef3a4-0138-4235-a963-df214ffcbd1d admin admin] PATH_INFO: `/v3/projects/3fa9f2503b0247cebe217bd8e141880a/users/2805c6cb77eb4063994ca851580019a7/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:18.400353 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57fef3a4-0138-4235-a963-df214ffcbd1d admin admin] RBAC: Authorizing `identity:create_grant(project_id=3fa9f2503b0247cebe217bd8e141880a, user_id=2805c6cb77eb4063994ca851580019a7, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:18.403629 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57fef3a4-0138-4235-a963-df214ffcbd1d admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:18.412993 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:18.424033 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 808/1623] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:01:18 2022] PUT /identity/v3/projects/3fa9f2503b0247cebe217bd8e141880a/users/2805c6cb77eb4063994ca851580019a7/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 67 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:18.425447 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-256345a7-4146-4367-94d4-d2d54913c0b7 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:18.436460 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-256345a7-4146-4367-94d4-d2d54913c0b7 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:18.510675 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b87681e-14c5-4c91-a39e-bf73884d9491 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:18.511352 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b87681e-14c5-4c91-a39e-bf73884d9491 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:18.511694 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b87681e-14c5-4c91-a39e-bf73884d9491 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:18.512537 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b87681e-14c5-4c91-a39e-bf73884d9491 admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:18.513200 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b87681e-14c5-4c91-a39e-bf73884d9491 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:18.515246 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b87681e-14c5-4c91-a39e-bf73884d9491 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:18.522641 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57fef3a4-0138-4235-a963-df214ffcbd1d admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:18.524579 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-57fef3a4-0138-4235-a963-df214ffcbd1d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:18.531782 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 816/1624] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:01:18 2022] GET /identity/v3/roles => generated 1420 bytes in 107 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:01:18.535811 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b87681e-14c5-4c91-a39e-bf73884d9491 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:18.546132 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b87681e-14c5-4c91-a39e-bf73884d9491 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:18.550741 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5e9ded1-1775-403f-aa8c-5d18e190d0e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:18.551387 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e9ded1-1775-403f-aa8c-5d18e190d0e2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:18.551660 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e9ded1-1775-403f-aa8c-5d18e190d0e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:18.551929 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e9ded1-1775-403f-aa8c-5d18e190d0e2 admin admin] PATH_INFO: `/v3/projects/3fa9f2503b0247cebe217bd8e141880a/users/2805c6cb77eb4063994ca851580019a7/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:18.553858 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5e9ded1-1775-403f-aa8c-5d18e190d0e2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3fa9f2503b0247cebe217bd8e141880a, user_id=2805c6cb77eb4063994ca851580019a7, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:18.557326 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5e9ded1-1775-403f-aa8c-5d18e190d0e2 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:18.569039 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 817/1625] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:01:18 2022] PUT /identity/v3/projects/3fa9f2503b0247cebe217bd8e141880a/users/2805c6cb77eb4063994ca851580019a7/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:01:18.570573 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5e9ded1-1775-403f-aa8c-5d18e190d0e2 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:18.572654 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-c5e9ded1-1775-403f-aa8c-5d18e190d0e2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:18.593373 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57fef3a4-0138-4235-a963-df214ffcbd1d admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:18.629481 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:18.644944 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ece6fc87-dfa5-49c1-a763-2cb3414ff406 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:18.645509 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece6fc87-dfa5-49c1-a763-2cb3414ff406 service cinder] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:18.645791 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece6fc87-dfa5-49c1-a763-2cb3414ff406 service cinder] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:18.646061 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece6fc87-dfa5-49c1-a763-2cb3414ff406 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:18.646239 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5e9ded1-1775-403f-aa8c-5d18e190d0e2 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:18.648040 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-ece6fc87-dfa5-49c1-a763-2cb3414ff406 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:18.651240 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ba8948d-8484-4ca2-a06e-5921b3f12cee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:18.651907 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba8948d-8484-4ca2-a06e-5921b3f12cee admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:18.652212 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba8948d-8484-4ca2-a06e-5921b3f12cee admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:18.652487 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba8948d-8484-4ca2-a06e-5921b3f12cee admin admin] PATH_INFO: `/v3/users/c185abe69a6e452dbfac0c057e64d369` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:18.653628 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ba8948d-8484-4ca2-a06e-5921b3f12cee admin admin] RBAC: Authorizing `identity:delete_user(user_id=c185abe69a6e452dbfac0c057e64d369)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:18.655596 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ba8948d-8484-4ca2-a06e-5921b3f12cee admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:18.719867 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-3ba8948d-8484-4ca2-a06e-5921b3f12cee admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c185abe69a6e452dbfac0c057e64d369'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:18.729305 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-3ba8948d-8484-4ca2-a06e-5921b3f12cee admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c185abe69a6e452dbfac0c057e64d369'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:18.729883 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ece6fc87-dfa5-49c1-a763-2cb3414ff406 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:18.731449 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ece6fc87-dfa5-49c1-a763-2cb3414ff406 service cinder] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:18.749290 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 809/1626] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:01:18 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 227 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 20 09:01:18.752614 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360faaa9-d37f-4dc8-ab04-f55b6a966d59 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:18.752857 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360faaa9-d37f-4dc8-ab04-f55b6a966d59 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:18.753129 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360faaa9-d37f-4dc8-ab04-f55b6a966d59 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:18.754755 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-3ba8948d-8484-4ca2-a06e-5921b3f12cee admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c185abe69a6e452dbfac0c057e64d369'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:18.770764 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-3ba8948d-8484-4ca2-a06e-5921b3f12cee admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c185abe69a6e452dbfac0c057e64d369'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:18.771598 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-3ba8948d-8484-4ca2-a06e-5921b3f12cee admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c185abe69a6e452dbfac0c057e64d369'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:18.784334 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-360faaa9-d37f-4dc8-ab04-f55b6a966d59 None None] MFA Rules not processed for user `2805c6cb77eb4063994ca851580019a7`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:18.787451 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 818/1627] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:01:18 2022] DELETE /identity/v3/users/c185abe69a6e452dbfac0c057e64d369 => generated 0 bytes in 215 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:18.787569 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-360faaa9-d37f-4dc8-ab04-f55b6a966d59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:18.789550 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-981f7290-bc41-46ec-9324-8865ff4e50db None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:18.789827 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-981f7290-bc41-46ec-9324-8865ff4e50db None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:18.790089 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-981f7290-bc41-46ec-9324-8865ff4e50db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:18.816255 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-981f7290-bc41-46ec-9324-8865ff4e50db None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:18.817472 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-981f7290-bc41-46ec-9324-8865ff4e50db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:18.878357 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:18.899480 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 810/1628] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:18 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:18.922122 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-360faaa9-d37f-4dc8-ab04-f55b6a966d59 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:18.924075 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-360faaa9-d37f-4dc8-ab04-f55b6a966d59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:18.948828 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 819/1629] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:18.961187 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-981f7290-bc41-46ec-9324-8865ff4e50db None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:18.963762 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-981f7290-bc41-46ec-9324-8865ff4e50db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:19.020674 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-360faaa9-d37f-4dc8-ab04-f55b6a966d59 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:19.029197 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a5596f0-74d1-490e-9189-9d8c0d27703c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:19.030070 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5596f0-74d1-490e-9189-9d8c0d27703c service cinder] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:19.030631 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5596f0-74d1-490e-9189-9d8c0d27703c service cinder] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:19.030912 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5596f0-74d1-490e-9189-9d8c0d27703c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:19.043239 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a5596f0-74d1-490e-9189-9d8c0d27703c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:19.045024 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a5596f0-74d1-490e-9189-9d8c0d27703c service cinder] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:19.057014 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 811/1630] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:01:18 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 135 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:01:19.069449 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-981f7290-bc41-46ec-9324-8865ff4e50db None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:19.073610 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43101d05-57cb-477a-8504-f6029486f8f3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:19.074393 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43101d05-57cb-477a-8504-f6029486f8f3 service neutron] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:19.074901 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43101d05-57cb-477a-8504-f6029486f8f3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:19.075320 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43101d05-57cb-477a-8504-f6029486f8f3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:19.088644 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43101d05-57cb-477a-8504-f6029486f8f3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:19.090518 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43101d05-57cb-477a-8504-f6029486f8f3 service neutron] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:19.108297 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 820/1631] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:01:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 149 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:01:19.750136 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a5596f0-74d1-490e-9189-9d8c0d27703c service cinder] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:19.752623 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-6a5596f0-74d1-490e-9189-9d8c0d27703c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:19.772235 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a5596f0-74d1-490e-9189-9d8c0d27703c service cinder] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:19.777017 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81175f9a-7b2b-4451-a413-1d63294cdd22 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:19.777715 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81175f9a-7b2b-4451-a413-1d63294cdd22 service cinder] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:19.778162 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81175f9a-7b2b-4451-a413-1d63294cdd22 service cinder] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:19.778535 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81175f9a-7b2b-4451-a413-1d63294cdd22 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:19.780429 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-81175f9a-7b2b-4451-a413-1d63294cdd22 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:19.925335 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81175f9a-7b2b-4451-a413-1d63294cdd22 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:19.928443 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81175f9a-7b2b-4451-a413-1d63294cdd22 service cinder] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:19.964804 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 812/1632] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:01:19 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 215 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 20 09:01:20.061078 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43101d05-57cb-477a-8504-f6029486f8f3 service neutron] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:20.062542 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-43101d05-57cb-477a-8504-f6029486f8f3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:20.083339 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43101d05-57cb-477a-8504-f6029486f8f3 service neutron] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:20.104835 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6e026b4-164b-4114-9aec-9f7d34075374 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:20.106546 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e026b4-164b-4114-9aec-9f7d34075374 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:20.106964 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e026b4-164b-4114-9aec-9f7d34075374 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:20.107338 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e026b4-164b-4114-9aec-9f7d34075374 admin admin] PATH_INFO: `/v3/projects/b40b02ee60bd47f9b2f3352911fe6aa4` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:20.110364 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6e026b4-164b-4114-9aec-9f7d34075374 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b40b02ee60bd47f9b2f3352911fe6aa4)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:20.121503 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6e026b4-164b-4114-9aec-9f7d34075374 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:20.151251 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:20.241575 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-f6e026b4-164b-4114-9aec-9f7d34075374 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b40b02ee60bd47f9b2f3352911fe6aa4', 'request_id': 'req-f6e026b4-164b-4114-9aec-9f7d34075374'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:20.250328 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-f6e026b4-164b-4114-9aec-9f7d34075374 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b40b02ee60bd47f9b2f3352911fe6aa4', 'request_id': 'req-f6e026b4-164b-4114-9aec-9f7d34075374'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:20.256761 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-f6e026b4-164b-4114-9aec-9f7d34075374 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b40b02ee60bd47f9b2f3352911fe6aa4', 'request_id': 'req-f6e026b4-164b-4114-9aec-9f7d34075374'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:20.277611 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-f6e026b4-164b-4114-9aec-9f7d34075374 admin admin] The token cache is being invalidate because project b40b02ee60bd47f9b2f3352911fe6aa4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81562) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:01:20.277911 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-f6e026b4-164b-4114-9aec-9f7d34075374 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:20.278781 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-f6e026b4-164b-4114-9aec-9f7d34075374 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:20.291803 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 821/1633] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:01:20 2022] DELETE /identity/v3/projects/b40b02ee60bd47f9b2f3352911fe6aa4 => generated 0 bytes in 231 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:20.295721 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81175f9a-7b2b-4451-a413-1d63294cdd22 service cinder] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:20.313090 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81175f9a-7b2b-4451-a413-1d63294cdd22 service cinder] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:20.412461 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43fc9a44-7cde-4a0f-9ed9-35719b02e719 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:20.413103 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43fc9a44-7cde-4a0f-9ed9-35719b02e719 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:20.413379 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43fc9a44-7cde-4a0f-9ed9-35719b02e719 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:20.414226 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43fc9a44-7cde-4a0f-9ed9-35719b02e719 admin admin] PATH_INFO: `/v3/users/b0c1d53f7c81422f83cd5a55efb8dfab` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:20.415538 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43fc9a44-7cde-4a0f-9ed9-35719b02e719 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b0c1d53f7c81422f83cd5a55efb8dfab)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:20.417513 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43fc9a44-7cde-4a0f-9ed9-35719b02e719 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:20.462779 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:20.481219 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-43fc9a44-7cde-4a0f-9ed9-35719b02e719 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b0c1d53f7c81422f83cd5a55efb8dfab'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:20.498786 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-43fc9a44-7cde-4a0f-9ed9-35719b02e719 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b0c1d53f7c81422f83cd5a55efb8dfab'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:20.524521 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-43fc9a44-7cde-4a0f-9ed9-35719b02e719 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b0c1d53f7c81422f83cd5a55efb8dfab'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:20.534018 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-43fc9a44-7cde-4a0f-9ed9-35719b02e719 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b0c1d53f7c81422f83cd5a55efb8dfab'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:20.534821 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-43fc9a44-7cde-4a0f-9ed9-35719b02e719 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b0c1d53f7c81422f83cd5a55efb8dfab'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:20.545680 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 813/1634] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:01:20 2022] DELETE /identity/v3/users/b0c1d53f7c81422f83cd5a55efb8dfab => generated 0 bytes in 248 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:20.815301 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6e026b4-164b-4114-9aec-9f7d34075374 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:20.816782 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-f6e026b4-164b-4114-9aec-9f7d34075374 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:20.844397 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6e026b4-164b-4114-9aec-9f7d34075374 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:20.907336 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-527ae785-4ae4-44af-ae2e-44fb6ccb0f03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:20.908138 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527ae785-4ae4-44af-ae2e-44fb6ccb0f03 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:20.908413 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527ae785-4ae4-44af-ae2e-44fb6ccb0f03 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:20.908760 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527ae785-4ae4-44af-ae2e-44fb6ccb0f03 admin admin] PATH_INFO: `/v3/projects/5847784d5e33418085e3a5be19a91d27` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:20.910150 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-527ae785-4ae4-44af-ae2e-44fb6ccb0f03 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5847784d5e33418085e3a5be19a91d27)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:20.912136 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-527ae785-4ae4-44af-ae2e-44fb6ccb0f03 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:20.985734 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-527ae785-4ae4-44af-ae2e-44fb6ccb0f03 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5847784d5e33418085e3a5be19a91d27', 'request_id': 'req-527ae785-4ae4-44af-ae2e-44fb6ccb0f03'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:20.992829 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-527ae785-4ae4-44af-ae2e-44fb6ccb0f03 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5847784d5e33418085e3a5be19a91d27', 'request_id': 'req-527ae785-4ae4-44af-ae2e-44fb6ccb0f03'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:21.001459 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-527ae785-4ae4-44af-ae2e-44fb6ccb0f03 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5847784d5e33418085e3a5be19a91d27', 'request_id': 'req-527ae785-4ae4-44af-ae2e-44fb6ccb0f03'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:21.015482 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-527ae785-4ae4-44af-ae2e-44fb6ccb0f03 admin admin] The token cache is being invalidate because project 5847784d5e33418085e3a5be19a91d27 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81562) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:01:21.015769 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-527ae785-4ae4-44af-ae2e-44fb6ccb0f03 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:21.016464 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-527ae785-4ae4-44af-ae2e-44fb6ccb0f03 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:21.018165 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 822/1635] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:01:20 2022] DELETE /identity/v3/projects/5847784d5e33418085e3a5be19a91d27 => generated 0 bytes in 203 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:21.023816 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c56d8ba2-5c62-4ffd-b588-97f1d9c029e2 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:21.024336 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c56d8ba2-5c62-4ffd-b588-97f1d9c029e2 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:21.024562 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c56d8ba2-5c62-4ffd-b588-97f1d9c029e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:21.048813 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-c56d8ba2-5c62-4ffd-b588-97f1d9c029e2 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:21.050065 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-c56d8ba2-5c62-4ffd-b588-97f1d9c029e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:21.122779 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-527ae785-4ae4-44af-ae2e-44fb6ccb0f03 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:21.122779 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-527ae785-4ae4-44af-ae2e-44fb6ccb0f03 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:21.123279 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 814/1636] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:21.125534 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02830d92-d48e-429d-a341-65ee29fb68ed None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:21.125825 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02830d92-d48e-429d-a341-65ee29fb68ed None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:21.126089 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02830d92-d48e-429d-a341-65ee29fb68ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:21.131439 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-527ae785-4ae4-44af-ae2e-44fb6ccb0f03 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:21.136777 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc78ee4-f818-42d7-b3ae-1fe5b6b16cb9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:21.137477 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc78ee4-f818-42d7-b3ae-1fe5b6b16cb9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:21.137823 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc78ee4-f818-42d7-b3ae-1fe5b6b16cb9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:21.138163 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc78ee4-f818-42d7-b3ae-1fe5b6b16cb9 admin admin] PATH_INFO: `/v3/users/c6407ae2e0fb4b2f8d729f804c672302` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:21.139430 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffc78ee4-f818-42d7-b3ae-1fe5b6b16cb9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c6407ae2e0fb4b2f8d729f804c672302)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:21.141438 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffc78ee4-f818-42d7-b3ae-1fe5b6b16cb9 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:21.148576 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-02830d92-d48e-429d-a341-65ee29fb68ed None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:21.149968 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-02830d92-d48e-429d-a341-65ee29fb68ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:21.158520 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 815/1637] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 20 09:01:21.162754 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02830d92-d48e-429d-a341-65ee29fb68ed None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:21.164456 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=82,keystone:DELETE=16,keystone:INSERT=2 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:21.172840 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02830d92-d48e-429d-a341-65ee29fb68ed None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:21.179718 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcecefb2-4c43-452c-853c-55b5fc167d79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:21.180243 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcecefb2-4c43-452c-853c-55b5fc167d79 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:21.180516 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcecefb2-4c43-452c-853c-55b5fc167d79 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:21.180780 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcecefb2-4c43-452c-853c-55b5fc167d79 admin admin] PATH_INFO: `/v3/domains` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:21.181420 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcecefb2-4c43-452c-853c-55b5fc167d79 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 09:01:21.181714 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcecefb2-4c43-452c-853c-55b5fc167d79 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:21.183446 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcecefb2-4c43-452c-853c-55b5fc167d79 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:21.196490 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 816/1638] 10.222.0.36 () {58 vars in 1153 bytes} [Thu Oct 20 09:01:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 09:01:21.198765 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-ffc78ee4-f818-42d7-b3ae-1fe5b6b16cb9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c6407ae2e0fb4b2f8d729f804c672302'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:21.200385 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcecefb2-4c43-452c-853c-55b5fc167d79 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:21.210462 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-ffc78ee4-f818-42d7-b3ae-1fe5b6b16cb9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c6407ae2e0fb4b2f8d729f804c672302'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:21.213683 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcecefb2-4c43-452c-853c-55b5fc167d79 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:21.224036 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb23da18-9843-4719-95a7-6a5bc841dc81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:21.224667 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb23da18-9843-4719-95a7-6a5bc841dc81 admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:21.224944 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb23da18-9843-4719-95a7-6a5bc841dc81 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:21.225212 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb23da18-9843-4719-95a7-6a5bc841dc81 admin admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:21.225870 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb23da18-9843-4719-95a7-6a5bc841dc81 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:21.228020 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb23da18-9843-4719-95a7-6a5bc841dc81 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:21.237503 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-ffc78ee4-f818-42d7-b3ae-1fe5b6b16cb9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c6407ae2e0fb4b2f8d729f804c672302'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:21.281154 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 817/1639] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:01:21 2022] POST /identity/v3/projects => generated 387 bytes in 78 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 20 09:01:21.281820 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-ffc78ee4-f818-42d7-b3ae-1fe5b6b16cb9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c6407ae2e0fb4b2f8d729f804c672302'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:21.283342 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb23da18-9843-4719-95a7-6a5bc841dc81 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:21.284355 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-ffc78ee4-f818-42d7-b3ae-1fe5b6b16cb9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c6407ae2e0fb4b2f8d729f804c672302'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:21.287838 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-eb23da18-9843-4719-95a7-6a5bc841dc81 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:21.300989 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 823/1640] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:01:21 2022] DELETE /identity/v3/users/c6407ae2e0fb4b2f8d729f804c672302 => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:21.306652 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2afe09be-7539-4e0b-a306-d8d28ce5fb8f None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:21.306942 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2afe09be-7539-4e0b-a306-d8d28ce5fb8f None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:21.307213 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2afe09be-7539-4e0b-a306-d8d28ce5fb8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:21.374866 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-2afe09be-7539-4e0b-a306-d8d28ce5fb8f None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:21.377415 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-2afe09be-7539-4e0b-a306-d8d28ce5fb8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:21.439137 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb23da18-9843-4719-95a7-6a5bc841dc81 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:21.478267 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=93,keystone:DELETE=5,keystone:INSERT=2 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:21.532270 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 824/1641] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 226 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 20 09:01:21.533758 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2afe09be-7539-4e0b-a306-d8d28ce5fb8f None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:21.535688 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-2afe09be-7539-4e0b-a306-d8d28ce5fb8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:21.584807 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a967d056-dbed-454f-a100-3a46e41f67e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:21.588126 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a967d056-dbed-454f-a100-3a46e41f67e5 admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:21.588772 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a967d056-dbed-454f-a100-3a46e41f67e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:21.589064 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a967d056-dbed-454f-a100-3a46e41f67e5 admin admin] PATH_INFO: `/v3/users` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:21.591041 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a967d056-dbed-454f-a100-3a46e41f67e5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:21.595901 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a967d056-dbed-454f-a100-3a46e41f67e5 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:21.661157 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 818/1642] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:01:21 2022] POST /identity/v3/users => generated 294 bytes in 377 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 20 09:01:21.662541 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a967d056-dbed-454f-a100-3a46e41f67e5 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:21.702642 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a967d056-dbed-454f-a100-3a46e41f67e5 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:21.712779 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2afe09be-7539-4e0b-a306-d8d28ce5fb8f None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:21.713683 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98a05bc4-2e06-4cf7-aa20-0ac7306528c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:21.714370 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a05bc4-2e06-4cf7-aa20-0ac7306528c7 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:21.714742 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a05bc4-2e06-4cf7-aa20-0ac7306528c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:21.715339 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a05bc4-2e06-4cf7-aa20-0ac7306528c7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:21.716953 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98a05bc4-2e06-4cf7-aa20-0ac7306528c7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:21.719808 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98a05bc4-2e06-4cf7-aa20-0ac7306528c7 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:21.729574 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c81fb08-c717-4bce-8c38-a447af725349 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:21.730142 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c81fb08-c717-4bce-8c38-a447af725349 service neutron] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:21.730422 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c81fb08-c717-4bce-8c38-a447af725349 service neutron] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:21.730740 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c81fb08-c717-4bce-8c38-a447af725349 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:21.743416 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 819/1643] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:01:21 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:01:21.744751 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98a05bc4-2e06-4cf7-aa20-0ac7306528c7 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:21.756465 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98a05bc4-2e06-4cf7-aa20-0ac7306528c7 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:21.762152 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c81fb08-c717-4bce-8c38-a447af725349 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:21.762900 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99d9fa18-60bc-4b00-a85c-d8481cdaf781 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:21.763866 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d9fa18-60bc-4b00-a85c-d8481cdaf781 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:21.764243 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d9fa18-60bc-4b00-a85c-d8481cdaf781 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:21.764751 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d9fa18-60bc-4b00-a85c-d8481cdaf781 admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:21.765797 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99d9fa18-60bc-4b00-a85c-d8481cdaf781 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:21.767565 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99d9fa18-60bc-4b00-a85c-d8481cdaf781 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:21.770205 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c81fb08-c717-4bce-8c38-a447af725349 service neutron] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:21.791349 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 820/1644] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:01:21 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:01:21.792762 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99d9fa18-60bc-4b00-a85c-d8481cdaf781 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:21.806059 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 825/1645] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:01:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 273 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:01:21.812955 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99d9fa18-60bc-4b00-a85c-d8481cdaf781 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:21.827902 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483801f3-1bcf-484b-949a-ad5b10dff7d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:21.829073 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483801f3-1bcf-484b-949a-ad5b10dff7d1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:21.829364 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483801f3-1bcf-484b-949a-ad5b10dff7d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:21.829642 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483801f3-1bcf-484b-949a-ad5b10dff7d1 admin admin] PATH_INFO: `/v3/projects/f9e1c6f0c4dc486c8c17a1ceab08bccd/users/dcdbac81ed524585894b17e78f8306d2/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:21.854190 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-483801f3-1bcf-484b-949a-ad5b10dff7d1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f9e1c6f0c4dc486c8c17a1ceab08bccd, user_id=dcdbac81ed524585894b17e78f8306d2, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:21.858344 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-483801f3-1bcf-484b-949a-ad5b10dff7d1 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:21.892405 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 821/1646] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:01:21 2022] PUT /identity/v3/projects/f9e1c6f0c4dc486c8c17a1ceab08bccd/users/dcdbac81ed524585894b17e78f8306d2/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:21.904591 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b504c39a-d0cc-494f-be33-0cd97cbbb2cc None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:21.904969 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b504c39a-d0cc-494f-be33-0cd97cbbb2cc None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:21.905320 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b504c39a-d0cc-494f-be33-0cd97cbbb2cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:21.934273 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-b504c39a-d0cc-494f-be33-0cd97cbbb2cc None None] MFA Rules not processed for user `dcdbac81ed524585894b17e78f8306d2`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:21.935526 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-b504c39a-d0cc-494f-be33-0cd97cbbb2cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:21.966981 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:DELETE=8,keystone:SELECT=91,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:22.010912 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 826/1647] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:21 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:22.015270 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483801f3-1bcf-484b-949a-ad5b10dff7d1 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:22.078289 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483801f3-1bcf-484b-949a-ad5b10dff7d1 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:22.082886 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbca2f36-36cd-49d7-9b7f-d72078d1826c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:22.083479 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbca2f36-36cd-49d7-9b7f-d72078d1826c admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:22.083752 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbca2f36-36cd-49d7-9b7f-d72078d1826c admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:22.084026 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbca2f36-36cd-49d7-9b7f-d72078d1826c admin admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:22.084640 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbca2f36-36cd-49d7-9b7f-d72078d1826c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:22.086608 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbca2f36-36cd-49d7-9b7f-d72078d1826c admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:22.113194 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b504c39a-d0cc-494f-be33-0cd97cbbb2cc None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:22.113912 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 822/1648] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:01:22 2022] POST /identity/v3/projects => generated 387 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:01:22.118014 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbca2f36-36cd-49d7-9b7f-d72078d1826c admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:22.118135 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-b504c39a-d0cc-494f-be33-0cd97cbbb2cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:22.136608 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b504c39a-d0cc-494f-be33-0cd97cbbb2cc None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:22.211892 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:22.225006 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7660c794-22e5-4ebf-b2e7-12aff173953b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:22.225777 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7660c794-22e5-4ebf-b2e7-12aff173953b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:22.226033 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7660c794-22e5-4ebf-b2e7-12aff173953b admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:22.226308 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7660c794-22e5-4ebf-b2e7-12aff173953b admin admin] PATH_INFO: `/v3/projects/26b9b0f59a9345418c8c423d277cfbae` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:22.227660 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7660c794-22e5-4ebf-b2e7-12aff173953b admin admin] RBAC: Authorizing `identity:delete_project(project_id=26b9b0f59a9345418c8c423d277cfbae)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:22.229440 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7660c794-22e5-4ebf-b2e7-12aff173953b admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:22.239346 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbca2f36-36cd-49d7-9b7f-d72078d1826c admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:22.246341 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a7b90b-89df-4165-8253-9d7df34fabc5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:22.247000 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a7b90b-89df-4165-8253-9d7df34fabc5 admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:22.247264 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a7b90b-89df-4165-8253-9d7df34fabc5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:22.247538 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a7b90b-89df-4165-8253-9d7df34fabc5 admin admin] PATH_INFO: `/v3/users` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:22.248180 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0a7b90b-89df-4165-8253-9d7df34fabc5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:22.250316 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0a7b90b-89df-4165-8253-9d7df34fabc5 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:22.280894 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 823/1649] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:01:22 2022] POST /identity/v3/users => generated 300 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 20 09:01:22.283062 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a7b90b-89df-4165-8253-9d7df34fabc5 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:22.300750 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-7660c794-22e5-4ebf-b2e7-12aff173953b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '26b9b0f59a9345418c8c423d277cfbae', 'request_id': 'req-7660c794-22e5-4ebf-b2e7-12aff173953b'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:22.301396 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a7b90b-89df-4165-8253-9d7df34fabc5 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:22.309508 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-7660c794-22e5-4ebf-b2e7-12aff173953b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '26b9b0f59a9345418c8c423d277cfbae', 'request_id': 'req-7660c794-22e5-4ebf-b2e7-12aff173953b'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:22.322337 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-7660c794-22e5-4ebf-b2e7-12aff173953b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '26b9b0f59a9345418c8c423d277cfbae', 'request_id': 'req-7660c794-22e5-4ebf-b2e7-12aff173953b'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:22.339676 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-7660c794-22e5-4ebf-b2e7-12aff173953b admin admin] The token cache is being invalidate because project 26b9b0f59a9345418c8c423d277cfbae was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81562) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:01:22.340370 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-7660c794-22e5-4ebf-b2e7-12aff173953b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:22.341135 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-7660c794-22e5-4ebf-b2e7-12aff173953b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:22.343948 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 827/1650] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:01:22 2022] DELETE /identity/v3/projects/26b9b0f59a9345418c8c423d277cfbae => generated 0 bytes in 231 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:01:22.347775 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7660c794-22e5-4ebf-b2e7-12aff173953b admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:22.350990 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-7660c794-22e5-4ebf-b2e7-12aff173953b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:22.404113 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31de6a52-9f45-4c72-8731-1be7a4ecd25e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:22.404740 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31de6a52-9f45-4c72-8731-1be7a4ecd25e admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:22.405019 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31de6a52-9f45-4c72-8731-1be7a4ecd25e admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:22.405418 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31de6a52-9f45-4c72-8731-1be7a4ecd25e admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:22.406250 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31de6a52-9f45-4c72-8731-1be7a4ecd25e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:22.408470 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31de6a52-9f45-4c72-8731-1be7a4ecd25e admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:22.422539 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 824/1651] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:01:22 2022] GET /identity/v3/roles => generated 1420 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:01:22.423816 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31de6a52-9f45-4c72-8731-1be7a4ecd25e admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:22.425463 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-31de6a52-9f45-4c72-8731-1be7a4ecd25e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:22.438181 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31de6a52-9f45-4c72-8731-1be7a4ecd25e admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:22.442991 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-404533e2-7e78-473c-be70-a4b069f8370d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:22.443633 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-404533e2-7e78-473c-be70-a4b069f8370d admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:22.443905 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-404533e2-7e78-473c-be70-a4b069f8370d admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:22.444170 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-404533e2-7e78-473c-be70-a4b069f8370d admin admin] PATH_INFO: `/v3/projects/c03a06d2e93a42be9b2279ac67b47dce/users/15ff9b4d1235455190995489643811f4/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:22.451284 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7660c794-22e5-4ebf-b2e7-12aff173953b admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:22.461695 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-404533e2-7e78-473c-be70-a4b069f8370d admin admin] RBAC: Authorizing `identity:create_grant(project_id=c03a06d2e93a42be9b2279ac67b47dce, user_id=15ff9b4d1235455190995489643811f4, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:22.463866 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c89024f0-8400-4bf4-b0c5-8d89f131d7d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:22.464541 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-404533e2-7e78-473c-be70-a4b069f8370d admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:22.464790 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c89024f0-8400-4bf4-b0c5-8d89f131d7d4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:22.465662 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c89024f0-8400-4bf4-b0c5-8d89f131d7d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:22.466044 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c89024f0-8400-4bf4-b0c5-8d89f131d7d4 admin admin] PATH_INFO: `/v3/users/2805c6cb77eb4063994ca851580019a7` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:22.467853 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c89024f0-8400-4bf4-b0c5-8d89f131d7d4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2805c6cb77eb4063994ca851580019a7)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:22.472784 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c89024f0-8400-4bf4-b0c5-8d89f131d7d4 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:22.474821 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:22.481080 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 825/1652] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:01:22 2022] PUT /identity/v3/projects/c03a06d2e93a42be9b2279ac67b47dce/users/15ff9b4d1235455190995489643811f4/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:01:22.484941 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-404533e2-7e78-473c-be70-a4b069f8370d admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:22.531676 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-c89024f0-8400-4bf4-b0c5-8d89f131d7d4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2805c6cb77eb4063994ca851580019a7'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:22.540806 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-c89024f0-8400-4bf4-b0c5-8d89f131d7d4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2805c6cb77eb4063994ca851580019a7'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:22.553111 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-404533e2-7e78-473c-be70-a4b069f8370d admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:22.557198 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-c89024f0-8400-4bf4-b0c5-8d89f131d7d4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2805c6cb77eb4063994ca851580019a7'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:22.559091 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8e32830-9102-4edd-a465-5515b4c625ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:22.560866 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8e32830-9102-4edd-a465-5515b4c625ac admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:22.561147 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8e32830-9102-4edd-a465-5515b4c625ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:22.561416 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8e32830-9102-4edd-a465-5515b4c625ac admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:22.562069 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8e32830-9102-4edd-a465-5515b4c625ac admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:22.563692 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8e32830-9102-4edd-a465-5515b4c625ac admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:22.565063 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-c89024f0-8400-4bf4-b0c5-8d89f131d7d4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2805c6cb77eb4063994ca851580019a7'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:22.566363 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-c89024f0-8400-4bf4-b0c5-8d89f131d7d4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2805c6cb77eb4063994ca851580019a7'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:22.572841 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 828/1653] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:01:22 2022] DELETE /identity/v3/users/2805c6cb77eb4063994ca851580019a7 => generated 0 bytes in 225 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:22.576920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 826/1654] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:01:22 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 20 09:01:22.580971 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c89024f0-8400-4bf4-b0c5-8d89f131d7d4 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:22.582666 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-c89024f0-8400-4bf4-b0c5-8d89f131d7d4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:22.656392 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c89024f0-8400-4bf4-b0c5-8d89f131d7d4 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:22.661553 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0906415b-b161-4bbe-91b6-dab83a65fc00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:22.662308 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0906415b-b161-4bbe-91b6-dab83a65fc00 admin admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:22.662685 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0906415b-b161-4bbe-91b6-dab83a65fc00 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:22.663017 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0906415b-b161-4bbe-91b6-dab83a65fc00 admin admin] PATH_INFO: `/v3/projects/c03a06d2e93a42be9b2279ac67b47dce/users/15ff9b4d1235455190995489643811f4/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:22.665632 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0906415b-b161-4bbe-91b6-dab83a65fc00 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c03a06d2e93a42be9b2279ac67b47dce, user_id=15ff9b4d1235455190995489643811f4, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:22.668621 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0906415b-b161-4bbe-91b6-dab83a65fc00 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:22.707129 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 829/1655] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:01:22 2022] PUT /identity/v3/projects/c03a06d2e93a42be9b2279ac67b47dce/users/15ff9b4d1235455190995489643811f4/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:01:22.709819 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a75ecde8-2dcb-4bde-8c11-0c642d63faa5 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:22.709819 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a75ecde8-2dcb-4bde-8c11-0c642d63faa5 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:22.709999 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a75ecde8-2dcb-4bde-8c11-0c642d63faa5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:22.744056 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-a75ecde8-2dcb-4bde-8c11-0c642d63faa5 None None] MFA Rules not processed for user `15ff9b4d1235455190995489643811f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:22.745186 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-a75ecde8-2dcb-4bde-8c11-0c642d63faa5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:22.781759 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:22.828217 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 827/1656] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:22 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:22.840821 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0906415b-b161-4bbe-91b6-dab83a65fc00 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:22.842914 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-0906415b-b161-4bbe-91b6-dab83a65fc00 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:22.929472 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0906415b-b161-4bbe-91b6-dab83a65fc00 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:22.935298 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df0c6275-cb44-4e5c-ad26-53ef0736b6c8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:22.935995 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df0c6275-cb44-4e5c-ad26-53ef0736b6c8 service cinder] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:22.936461 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df0c6275-cb44-4e5c-ad26-53ef0736b6c8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:22.936831 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df0c6275-cb44-4e5c-ad26-53ef0736b6c8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:22.948956 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df0c6275-cb44-4e5c-ad26-53ef0736b6c8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:22.951575 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a75ecde8-2dcb-4bde-8c11-0c642d63faa5 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:22.953215 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-a75ecde8-2dcb-4bde-8c11-0c642d63faa5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:22.956553 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df0c6275-cb44-4e5c-ad26-53ef0736b6c8 service cinder] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:22.973422 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 830/1657] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:01:22 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 133 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:01:23.098630 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a75ecde8-2dcb-4bde-8c11-0c642d63faa5 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:23.112543 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e6e3b15-ec48-416b-aba7-4180179d5b63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:23.113202 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6e3b15-ec48-416b-aba7-4180179d5b63 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:23.113481 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6e3b15-ec48-416b-aba7-4180179d5b63 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:23.113755 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6e3b15-ec48-416b-aba7-4180179d5b63 admin admin] PATH_INFO: `/v3/projects/3fa9f2503b0247cebe217bd8e141880a` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:23.117351 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e6e3b15-ec48-416b-aba7-4180179d5b63 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3fa9f2503b0247cebe217bd8e141880a)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:23.120023 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e6e3b15-ec48-416b-aba7-4180179d5b63 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:23.241221 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-2e6e3b15-ec48-416b-aba7-4180179d5b63 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3fa9f2503b0247cebe217bd8e141880a', 'request_id': 'req-2e6e3b15-ec48-416b-aba7-4180179d5b63'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:23.249270 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-2e6e3b15-ec48-416b-aba7-4180179d5b63 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3fa9f2503b0247cebe217bd8e141880a', 'request_id': 'req-2e6e3b15-ec48-416b-aba7-4180179d5b63'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:23.257808 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-2e6e3b15-ec48-416b-aba7-4180179d5b63 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3fa9f2503b0247cebe217bd8e141880a', 'request_id': 'req-2e6e3b15-ec48-416b-aba7-4180179d5b63'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:23.277434 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-2e6e3b15-ec48-416b-aba7-4180179d5b63 admin admin] The token cache is being invalidate because project 3fa9f2503b0247cebe217bd8e141880a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81561) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:01:23.278052 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-2e6e3b15-ec48-416b-aba7-4180179d5b63 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:23.279507 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-2e6e3b15-ec48-416b-aba7-4180179d5b63 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:23.282278 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 828/1658] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:01:22 2022] DELETE /identity/v3/projects/3fa9f2503b0247cebe217bd8e141880a => generated 0 bytes in 331 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:01:23.286319 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a587b7f-3338-4e6e-be82-102f19299be1 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:23.286844 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a587b7f-3338-4e6e-be82-102f19299be1 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:23.286924 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a587b7f-3338-4e6e-be82-102f19299be1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:23.327059 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-7a587b7f-3338-4e6e-be82-102f19299be1 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:23.328344 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-7a587b7f-3338-4e6e-be82-102f19299be1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:23.347405 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:23.421614 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 831/1659] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:23.429802 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a5d4cc-dab2-493d-a42a-6cb34288b2db None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:23.430373 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a5d4cc-dab2-493d-a42a-6cb34288b2db None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:23.430816 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a5d4cc-dab2-493d-a42a-6cb34288b2db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:23.456567 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-a7a5d4cc-dab2-493d-a42a-6cb34288b2db None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:23.458362 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-a7a5d4cc-dab2-493d-a42a-6cb34288b2db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:23.468732 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 829/1660] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:23.472794 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a587b7f-3338-4e6e-be82-102f19299be1 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:23.483531 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a587b7f-3338-4e6e-be82-102f19299be1 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:23.504261 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daf1193a-2958-4c75-a2aa-54a5663539a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:23.504886 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf1193a-2958-4c75-a2aa-54a5663539a5 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:23.505161 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf1193a-2958-4c75-a2aa-54a5663539a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:23.505432 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf1193a-2958-4c75-a2aa-54a5663539a5 admin admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:23.506570 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daf1193a-2958-4c75-a2aa-54a5663539a5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 09:01:23.506570 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daf1193a-2958-4c75-a2aa-54a5663539a5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:23.509343 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daf1193a-2958-4c75-a2aa-54a5663539a5 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:23.521567 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 832/1661] 10.222.0.36 () {58 vars in 1153 bytes} [Thu Oct 20 09:01:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 09:01:23.526157 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7a5d4cc-dab2-493d-a42a-6cb34288b2db None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:23.535887 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7a5d4cc-dab2-493d-a42a-6cb34288b2db None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:23.541350 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb8834fc-d00b-4d82-a6fe-6a2c8c3508d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:23.542129 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb8834fc-d00b-4d82-a6fe-6a2c8c3508d0 admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:23.542411 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb8834fc-d00b-4d82-a6fe-6a2c8c3508d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:23.542709 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb8834fc-d00b-4d82-a6fe-6a2c8c3508d0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:23.543354 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb8834fc-d00b-4d82-a6fe-6a2c8c3508d0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:23.545715 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb8834fc-d00b-4d82-a6fe-6a2c8c3508d0 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:23.579794 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 830/1662] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:01:23 2022] POST /identity/v3/projects => generated 409 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 20 09:01:23.583324 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daf1193a-2958-4c75-a2aa-54a5663539a5 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:23.598656 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daf1193a-2958-4c75-a2aa-54a5663539a5 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:23.693699 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8391d83-9018-4e9c-8b0d-b94d3b714edb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:23.694302 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8391d83-9018-4e9c-8b0d-b94d3b714edb admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:23.694611 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8391d83-9018-4e9c-8b0d-b94d3b714edb admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:23.694915 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8391d83-9018-4e9c-8b0d-b94d3b714edb admin admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:23.695607 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8391d83-9018-4e9c-8b0d-b94d3b714edb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:23.697575 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8391d83-9018-4e9c-8b0d-b94d3b714edb admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:23.725801 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 833/1663] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:01:23 2022] POST /identity/v3/users => generated 305 bytes in 140 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 20 09:01:23.727481 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb8834fc-d00b-4d82-a6fe-6a2c8c3508d0 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:23.738144 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb8834fc-d00b-4d82-a6fe-6a2c8c3508d0 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:23.742988 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b8ac09e-6131-4478-9cec-1ca12327c382 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:23.743687 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8ac09e-6131-4478-9cec-1ca12327c382 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:23.743967 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8ac09e-6131-4478-9cec-1ca12327c382 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:23.744233 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8ac09e-6131-4478-9cec-1ca12327c382 admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:23.744870 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b8ac09e-6131-4478-9cec-1ca12327c382 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:23.746720 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b8ac09e-6131-4478-9cec-1ca12327c382 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:23.763029 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 831/1664] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:01:23 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:01:23.764013 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8391d83-9018-4e9c-8b0d-b94d3b714edb admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:23.774291 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8391d83-9018-4e9c-8b0d-b94d3b714edb admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:23.778886 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a95b0b34-a483-41b5-a179-898c897bafc6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:23.779471 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95b0b34-a483-41b5-a179-898c897bafc6 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:23.779743 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95b0b34-a483-41b5-a179-898c897bafc6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:23.780011 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95b0b34-a483-41b5-a179-898c897bafc6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:23.780669 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a95b0b34-a483-41b5-a179-898c897bafc6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:23.782249 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a95b0b34-a483-41b5-a179-898c897bafc6 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:23.793230 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 834/1665] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:01:23 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 20 09:01:23.797713 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b8ac09e-6131-4478-9cec-1ca12327c382 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:23.811863 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b8ac09e-6131-4478-9cec-1ca12327c382 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:23.816801 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af7da188-4b90-442e-9fcf-5bb684bdb4eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:23.817650 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af7da188-4b90-442e-9fcf-5bb684bdb4eb admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:23.817777 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af7da188-4b90-442e-9fcf-5bb684bdb4eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:23.818117 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af7da188-4b90-442e-9fcf-5bb684bdb4eb admin admin] PATH_INFO: `/v3/projects/cc24996ae3c240a0a8412a3b526d0f4a/users/06e79ff08f0b45d1aedad0824bb0c62f/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:23.826123 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a95b0b34-a483-41b5-a179-898c897bafc6 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:23.828127 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-a95b0b34-a483-41b5-a179-898c897bafc6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:23.836068 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af7da188-4b90-442e-9fcf-5bb684bdb4eb admin admin] RBAC: Authorizing `identity:create_grant(project_id=cc24996ae3c240a0a8412a3b526d0f4a, user_id=06e79ff08f0b45d1aedad0824bb0c62f, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:23.838875 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af7da188-4b90-442e-9fcf-5bb684bdb4eb admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:23.841572 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:23.845267 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a95b0b34-a483-41b5-a179-898c897bafc6 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:23.850257 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09ea0bc5-dca7-4351-bef6-a44a909a42f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:23.851767 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ea0bc5-dca7-4351-bef6-a44a909a42f8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:23.852457 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ea0bc5-dca7-4351-bef6-a44a909a42f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:23.852730 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ea0bc5-dca7-4351-bef6-a44a909a42f8 admin admin] PATH_INFO: `/v3/users/dcdbac81ed524585894b17e78f8306d2` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:23.854510 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09ea0bc5-dca7-4351-bef6-a44a909a42f8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=dcdbac81ed524585894b17e78f8306d2)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:23.856183 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09ea0bc5-dca7-4351-bef6-a44a909a42f8 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:23.860970 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 832/1666] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:01:23 2022] PUT /identity/v3/projects/cc24996ae3c240a0a8412a3b526d0f4a/users/06e79ff08f0b45d1aedad0824bb0c62f/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:23.871799 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c674eb2-2814-4ccd-81ad-9558d578bcee None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:23.872602 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c674eb2-2814-4ccd-81ad-9558d578bcee None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:23.872883 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c674eb2-2814-4ccd-81ad-9558d578bcee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:23.909901 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-8c674eb2-2814-4ccd-81ad-9558d578bcee None None] MFA Rules not processed for user `06e79ff08f0b45d1aedad0824bb0c62f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:23.911039 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-8c674eb2-2814-4ccd-81ad-9558d578bcee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:23.916520 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-09ea0bc5-dca7-4351-bef6-a44a909a42f8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dcdbac81ed524585894b17e78f8306d2'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:23.927221 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-09ea0bc5-dca7-4351-bef6-a44a909a42f8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dcdbac81ed524585894b17e78f8306d2'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:23.940884 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-09ea0bc5-dca7-4351-bef6-a44a909a42f8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dcdbac81ed524585894b17e78f8306d2'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:23.947108 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=86,keystone:INSERT=4,keystone:DELETE=10 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:23.949597 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-09ea0bc5-dca7-4351-bef6-a44a909a42f8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dcdbac81ed524585894b17e78f8306d2'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:23.950752 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-09ea0bc5-dca7-4351-bef6-a44a909a42f8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dcdbac81ed524585894b17e78f8306d2'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:23.960326 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 835/1667] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:01:23 2022] DELETE /identity/v3/users/dcdbac81ed524585894b17e78f8306d2 => generated 0 bytes in 134 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:01:23.966616 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b5fe1a1-d49e-4ec7-8903-3ae5db4a02a2 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:23.966988 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b5fe1a1-d49e-4ec7-8903-3ae5db4a02a2 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:23.967357 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b5fe1a1-d49e-4ec7-8903-3ae5db4a02a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:23.989129 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-9b5fe1a1-d49e-4ec7-8903-3ae5db4a02a2 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:23.990391 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-9b5fe1a1-d49e-4ec7-8903-3ae5db4a02a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:24.002997 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 833/1668] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:23 2022] POST /identity/v3/auth/tokens => generated 3286 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 20 09:01:24.008758 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c674eb2-2814-4ccd-81ad-9558d578bcee None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:24.010373 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-8c674eb2-2814-4ccd-81ad-9558d578bcee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:24.064105 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 836/1669] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:24.074769 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b5fe1a1-d49e-4ec7-8903-3ae5db4a02a2 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:24.076378 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-9b5fe1a1-d49e-4ec7-8903-3ae5db4a02a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:24.090568 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c674eb2-2814-4ccd-81ad-9558d578bcee None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:24.095437 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8ec557a-d9d5-4740-b166-05f3faf4523c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:24.096048 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ec557a-d9d5-4740-b166-05f3faf4523c admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:24.096327 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ec557a-d9d5-4740-b166-05f3faf4523c admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:24.096591 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ec557a-d9d5-4740-b166-05f3faf4523c admin admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:24.097288 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8ec557a-d9d5-4740-b166-05f3faf4523c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:24.099235 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8ec557a-d9d5-4740-b166-05f3faf4523c admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:24.134513 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 834/1670] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:01:24 2022] POST /identity/v3/projects => generated 409 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 20 09:01:24.137210 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8ec557a-d9d5-4740-b166-05f3faf4523c admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:24.141059 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b5fe1a1-d49e-4ec7-8903-3ae5db4a02a2 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:24.200172 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fabee555-8a92-4d35-9281-c3994acfcce2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:24.200786 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fabee555-8a92-4d35-9281-c3994acfcce2 service neutron] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:24.201060 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fabee555-8a92-4d35-9281-c3994acfcce2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:24.201324 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fabee555-8a92-4d35-9281-c3994acfcce2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:24.217375 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8ec557a-d9d5-4740-b166-05f3faf4523c admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:24.224300 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02e9bc1d-e3c1-478d-924a-62eaf2398c2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:24.225203 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e9bc1d-e3c1-478d-924a-62eaf2398c2d admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:24.225642 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e9bc1d-e3c1-478d-924a-62eaf2398c2d admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:24.225642 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e9bc1d-e3c1-478d-924a-62eaf2398c2d admin admin] PATH_INFO: `/v3/users` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:24.226296 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02e9bc1d-e3c1-478d-924a-62eaf2398c2d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:24.228557 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02e9bc1d-e3c1-478d-924a-62eaf2398c2d admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:24.249309 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:INSERT=5,keystone:SELECT=95 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:24.261028 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 835/1671] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:01:24 2022] POST /identity/v3/users => generated 311 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:01:24.262700 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02e9bc1d-e3c1-478d-924a-62eaf2398c2d admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:24.275644 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02e9bc1d-e3c1-478d-924a-62eaf2398c2d admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:24.284068 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e67d5482-d55e-4582-b821-e74f5e310c28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:24.285597 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e67d5482-d55e-4582-b821-e74f5e310c28 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:24.285922 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e67d5482-d55e-4582-b821-e74f5e310c28 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:24.286241 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e67d5482-d55e-4582-b821-e74f5e310c28 admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:24.287677 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e67d5482-d55e-4582-b821-e74f5e310c28 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:24.288807 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fabee555-8a92-4d35-9281-c3994acfcce2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:24.289599 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e67d5482-d55e-4582-b821-e74f5e310c28 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:24.290563 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fabee555-8a92-4d35-9281-c3994acfcce2 service neutron] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:24.300138 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:24.302591 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 836/1672] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:01:24 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:01:24.307461 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e67d5482-d55e-4582-b821-e74f5e310c28 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:24.310134 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 837/1673] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:01:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 236 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 20 09:01:24.324738 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e67d5482-d55e-4582-b821-e74f5e310c28 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:24.342764 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3ccd389-bec6-4633-8bac-1eee89e465fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:24.343816 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3ccd389-bec6-4633-8bac-1eee89e465fa admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:24.344101 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3ccd389-bec6-4633-8bac-1eee89e465fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:24.344374 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3ccd389-bec6-4633-8bac-1eee89e465fa admin admin] PATH_INFO: `/v3/projects/39cc92ab78a142629dff196154f19c5c/users/c0087161752b4c789e34d0c40b420483/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:24.386913 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3ccd389-bec6-4633-8bac-1eee89e465fa admin admin] RBAC: Authorizing `identity:create_grant(project_id=39cc92ab78a142629dff196154f19c5c, user_id=c0087161752b4c789e34d0c40b420483, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:24.408622 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3ccd389-bec6-4633-8bac-1eee89e465fa admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:24.448185 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 837/1674] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:01:24 2022] PUT /identity/v3/projects/39cc92ab78a142629dff196154f19c5c/users/c0087161752b4c789e34d0c40b420483/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 139 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:01:24.450437 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fabee555-8a92-4d35-9281-c3994acfcce2 service neutron] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:24.483096 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fabee555-8a92-4d35-9281-c3994acfcce2 service neutron] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:24.598265 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21550bb7-e95a-4168-87ae-b67a043f146e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:24.598911 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21550bb7-e95a-4168-87ae-b67a043f146e admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:24.599196 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21550bb7-e95a-4168-87ae-b67a043f146e admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:24.599456 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21550bb7-e95a-4168-87ae-b67a043f146e admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:24.600095 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21550bb7-e95a-4168-87ae-b67a043f146e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:24.601728 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21550bb7-e95a-4168-87ae-b67a043f146e admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:24.612270 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 838/1675] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:01:24 2022] GET /identity/v3/roles => generated 1420 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:01:24.616287 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3ccd389-bec6-4633-8bac-1eee89e465fa admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:24.628397 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3ccd389-bec6-4633-8bac-1eee89e465fa admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:24.642534 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3c85a7f-93da-481a-a6b5-dbaac832d163 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:24.644235 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3c85a7f-93da-481a-a6b5-dbaac832d163 admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:24.644603 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3c85a7f-93da-481a-a6b5-dbaac832d163 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:24.645142 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3c85a7f-93da-481a-a6b5-dbaac832d163 admin admin] PATH_INFO: `/v3/projects/39cc92ab78a142629dff196154f19c5c/users/c0087161752b4c789e34d0c40b420483/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:24.647481 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3c85a7f-93da-481a-a6b5-dbaac832d163 admin admin] RBAC: Authorizing `identity:create_grant(project_id=39cc92ab78a142629dff196154f19c5c, user_id=c0087161752b4c789e34d0c40b420483, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:24.650427 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3c85a7f-93da-481a-a6b5-dbaac832d163 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:24.661742 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 838/1676] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:01:24 2022] PUT /identity/v3/projects/39cc92ab78a142629dff196154f19c5c/users/c0087161752b4c789e34d0c40b420483/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:24.674015 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b360c6fc-1425-47ab-9a4d-0489ba2f36f2 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:24.674298 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b360c6fc-1425-47ab-9a4d-0489ba2f36f2 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:24.674657 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b360c6fc-1425-47ab-9a4d-0489ba2f36f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:24.709050 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-b360c6fc-1425-47ab-9a4d-0489ba2f36f2 None None] MFA Rules not processed for user `c0087161752b4c789e34d0c40b420483`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:24.710286 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-b360c6fc-1425-47ab-9a4d-0489ba2f36f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:24.747832 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3c85a7f-93da-481a-a6b5-dbaac832d163 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:24.749577 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-c3c85a7f-93da-481a-a6b5-dbaac832d163 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:24.795386 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 839/1677] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:24 2022] POST /identity/v3/auth/tokens => generated 3353 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:24.808343 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b360c6fc-1425-47ab-9a4d-0489ba2f36f2 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:24.810960 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-b360c6fc-1425-47ab-9a4d-0489ba2f36f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:24.823449 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3c85a7f-93da-481a-a6b5-dbaac832d163 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:24.828340 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b204e72a-3d92-4b0c-98b0-a305f4164288 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:24.829032 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b204e72a-3d92-4b0c-98b0-a305f4164288 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:24.829319 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b204e72a-3d92-4b0c-98b0-a305f4164288 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:24.829703 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b204e72a-3d92-4b0c-98b0-a305f4164288 admin admin] PATH_INFO: `/v3/users/de576ca2e5cc462c910eec4e435f7ac7` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:24.830942 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b204e72a-3d92-4b0c-98b0-a305f4164288 admin admin] RBAC: Authorizing `identity:delete_user(user_id=de576ca2e5cc462c910eec4e435f7ac7)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:24.833025 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b204e72a-3d92-4b0c-98b0-a305f4164288 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:24.895606 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b360c6fc-1425-47ab-9a4d-0489ba2f36f2 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:24.899406 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-974237ca-c53d-4c8a-af09-16871b91c193 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:24.900017 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974237ca-c53d-4c8a-af09-16871b91c193 service cinder] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:24.900294 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974237ca-c53d-4c8a-af09-16871b91c193 service cinder] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:24.900561 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974237ca-c53d-4c8a-af09-16871b91c193 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:24.901002 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-b204e72a-3d92-4b0c-98b0-a305f4164288 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'de576ca2e5cc462c910eec4e435f7ac7'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:24.911244 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-974237ca-c53d-4c8a-af09-16871b91c193 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:24.912395 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-b204e72a-3d92-4b0c-98b0-a305f4164288 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'de576ca2e5cc462c910eec4e435f7ac7'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:24.912682 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-974237ca-c53d-4c8a-af09-16871b91c193 service cinder] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:24.923699 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 840/1678] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:01:24 2022] GET /identity/v3/auth/tokens => generated 3353 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 20 09:01:24.924578 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-974237ca-c53d-4c8a-af09-16871b91c193 service cinder] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:24.926899 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-974237ca-c53d-4c8a-af09-16871b91c193 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:24.940039 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-b204e72a-3d92-4b0c-98b0-a305f4164288 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'de576ca2e5cc462c910eec4e435f7ac7'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:24.944898 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-974237ca-c53d-4c8a-af09-16871b91c193 service cinder] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:24.947836 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-b204e72a-3d92-4b0c-98b0-a305f4164288 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'de576ca2e5cc462c910eec4e435f7ac7'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:24.948491 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-b204e72a-3d92-4b0c-98b0-a305f4164288 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'de576ca2e5cc462c910eec4e435f7ac7'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:24.950285 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-653b1d38-daad-4e7d-baf2-322917760625 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:24.950944 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-653b1d38-daad-4e7d-baf2-322917760625 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:24.951227 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-653b1d38-daad-4e7d-baf2-322917760625 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:24.951511 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-653b1d38-daad-4e7d-baf2-322917760625 admin admin] PATH_INFO: `/v3/projects/f9e1c6f0c4dc486c8c17a1ceab08bccd` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:24.952672 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-653b1d38-daad-4e7d-baf2-322917760625 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f9e1c6f0c4dc486c8c17a1ceab08bccd)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:24.954952 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-653b1d38-daad-4e7d-baf2-322917760625 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:24.955591 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 839/1679] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:01:24 2022] DELETE /identity/v3/users/de576ca2e5cc462c910eec4e435f7ac7 => generated 0 bytes in 208 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:01:24.960597 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67e1a45-e3aa-4e36-96f4-b66735f5d3a3 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:24.960885 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67e1a45-e3aa-4e36-96f4-b66735f5d3a3 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:24.961186 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67e1a45-e3aa-4e36-96f4-b66735f5d3a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:24.984329 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-a67e1a45-e3aa-4e36-96f4-b66735f5d3a3 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:24.984899 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:24.987004 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-a67e1a45-e3aa-4e36-96f4-b66735f5d3a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:25.013863 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=87,keystone:INSERT=3,keystone:DELETE=10 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:25.032748 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-653b1d38-daad-4e7d-baf2-322917760625 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f9e1c6f0c4dc486c8c17a1ceab08bccd', 'request_id': 'req-653b1d38-daad-4e7d-baf2-322917760625'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:25.041812 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-653b1d38-daad-4e7d-baf2-322917760625 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f9e1c6f0c4dc486c8c17a1ceab08bccd', 'request_id': 'req-653b1d38-daad-4e7d-baf2-322917760625'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:25.048342 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-653b1d38-daad-4e7d-baf2-322917760625 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f9e1c6f0c4dc486c8c17a1ceab08bccd', 'request_id': 'req-653b1d38-daad-4e7d-baf2-322917760625'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:25.063010 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-653b1d38-daad-4e7d-baf2-322917760625 admin admin] The token cache is being invalidate because project f9e1c6f0c4dc486c8c17a1ceab08bccd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81562) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:01:25.063308 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-653b1d38-daad-4e7d-baf2-322917760625 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:25.063954 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-653b1d38-daad-4e7d-baf2-322917760625 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:25.066374 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 840/1681] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 20 09:01:25.068511 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 841/1681] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:01:24 2022] DELETE /identity/v3/projects/f9e1c6f0c4dc486c8c17a1ceab08bccd => generated 0 bytes in 141 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:25.069549 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a67e1a45-e3aa-4e36-96f4-b66735f5d3a3 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:25.071504 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-a67e1a45-e3aa-4e36-96f4-b66735f5d3a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:25.076739 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-653b1d38-daad-4e7d-baf2-322917760625 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:25.078127 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-653b1d38-daad-4e7d-baf2-322917760625 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:25.091133 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a67e1a45-e3aa-4e36-96f4-b66735f5d3a3 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:25.143919 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-653b1d38-daad-4e7d-baf2-322917760625 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:25.148886 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63cb2a40-69da-4604-a4c3-233b1fa58aed service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:25.149517 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63cb2a40-69da-4604-a4c3-233b1fa58aed service neutron] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:25.149781 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63cb2a40-69da-4604-a4c3-233b1fa58aed service neutron] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:25.150056 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63cb2a40-69da-4604-a4c3-233b1fa58aed service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:25.151831 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-63cb2a40-69da-4604-a4c3-233b1fa58aed service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:25.156551 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06f93d4d-e264-4353-82fc-f761e0121a1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:25.157154 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f93d4d-e264-4353-82fc-f761e0121a1a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:25.157435 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f93d4d-e264-4353-82fc-f761e0121a1a admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:25.157709 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f93d4d-e264-4353-82fc-f761e0121a1a admin admin] PATH_INFO: `/v3/users/15ff9b4d1235455190995489643811f4` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:25.158915 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06f93d4d-e264-4353-82fc-f761e0121a1a admin admin] RBAC: Authorizing `identity:delete_user(user_id=15ff9b4d1235455190995489643811f4)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:25.160859 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06f93d4d-e264-4353-82fc-f761e0121a1a admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:25.166171 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63cb2a40-69da-4604-a4c3-233b1fa58aed service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:25.168251 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63cb2a40-69da-4604-a4c3-233b1fa58aed service neutron] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:25.180012 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 842/1682] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:01:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 20 09:01:25.180947 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63cb2a40-69da-4604-a4c3-233b1fa58aed service neutron] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:25.182713 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-63cb2a40-69da-4604-a4c3-233b1fa58aed service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:25.255601 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-06f93d4d-e264-4353-82fc-f761e0121a1a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '15ff9b4d1235455190995489643811f4'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:25.271214 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-06f93d4d-e264-4353-82fc-f761e0121a1a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '15ff9b4d1235455190995489643811f4'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:25.294407 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-06f93d4d-e264-4353-82fc-f761e0121a1a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '15ff9b4d1235455190995489643811f4'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:25.297752 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63cb2a40-69da-4604-a4c3-233b1fa58aed service neutron] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:25.303995 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c87f77b-2b2a-4f6d-8cd7-5d8dbc73361d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:25.304616 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-06f93d4d-e264-4353-82fc-f761e0121a1a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '15ff9b4d1235455190995489643811f4'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:25.304734 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c87f77b-2b2a-4f6d-8cd7-5d8dbc73361d service cinder] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:25.305026 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c87f77b-2b2a-4f6d-8cd7-5d8dbc73361d service cinder] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:25.305600 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c87f77b-2b2a-4f6d-8cd7-5d8dbc73361d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:25.306007 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-06f93d4d-e264-4353-82fc-f761e0121a1a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '15ff9b4d1235455190995489643811f4'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:25.307710 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-7c87f77b-2b2a-4f6d-8cd7-5d8dbc73361d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:25.314518 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 841/1683] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:01:25 2022] DELETE /identity/v3/users/15ff9b4d1235455190995489643811f4 => generated 0 bytes in 245 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:01:25.389368 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c87f77b-2b2a-4f6d-8cd7-5d8dbc73361d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:25.390930 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c87f77b-2b2a-4f6d-8cd7-5d8dbc73361d service cinder] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:25.401125 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 843/1684] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:01:25 2022] GET /identity/v3/auth/tokens => generated 3286 bytes in 220 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:01:25.600161 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06f93d4d-e264-4353-82fc-f761e0121a1a admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:25.601871 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-06f93d4d-e264-4353-82fc-f761e0121a1a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:25.757981 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:25.796053 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06f93d4d-e264-4353-82fc-f761e0121a1a admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:25.808248 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f8d1ba6-734b-4c0f-a987-d2559ac98be9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:25.810203 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8d1ba6-734b-4c0f-a987-d2559ac98be9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:25.814254 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8d1ba6-734b-4c0f-a987-d2559ac98be9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:25.814731 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8d1ba6-734b-4c0f-a987-d2559ac98be9 admin admin] PATH_INFO: `/v3/projects/192b294d0db04fbcaf3b125da598ea2b` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:25.816739 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f8d1ba6-734b-4c0f-a987-d2559ac98be9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=192b294d0db04fbcaf3b125da598ea2b)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:25.820218 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f8d1ba6-734b-4c0f-a987-d2559ac98be9 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:25.917003 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-9f8d1ba6-734b-4c0f-a987-d2559ac98be9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '192b294d0db04fbcaf3b125da598ea2b', 'request_id': 'req-9f8d1ba6-734b-4c0f-a987-d2559ac98be9'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:25.926857 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-9f8d1ba6-734b-4c0f-a987-d2559ac98be9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '192b294d0db04fbcaf3b125da598ea2b', 'request_id': 'req-9f8d1ba6-734b-4c0f-a987-d2559ac98be9'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:25.941611 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-9f8d1ba6-734b-4c0f-a987-d2559ac98be9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '192b294d0db04fbcaf3b125da598ea2b', 'request_id': 'req-9f8d1ba6-734b-4c0f-a987-d2559ac98be9'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:25.950158 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c87f77b-2b2a-4f6d-8cd7-5d8dbc73361d service cinder] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:25.951776 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-7c87f77b-2b2a-4f6d-8cd7-5d8dbc73361d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:25.966416 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:25.975815 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c87f77b-2b2a-4f6d-8cd7-5d8dbc73361d service cinder] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:25.978460 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-9f8d1ba6-734b-4c0f-a987-d2559ac98be9 admin admin] The token cache is being invalidate because project 192b294d0db04fbcaf3b125da598ea2b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81561) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:01:25.978803 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-9f8d1ba6-734b-4c0f-a987-d2559ac98be9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:25.980254 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-9f8d1ba6-734b-4c0f-a987-d2559ac98be9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:25.982515 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 842/1685] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:01:25 2022] DELETE /identity/v3/projects/192b294d0db04fbcaf3b125da598ea2b => generated 0 bytes in 383 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:25.987323 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f8d1ba6-734b-4c0f-a987-d2559ac98be9 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:25.993059 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-9f8d1ba6-734b-4c0f-a987-d2559ac98be9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:26.105500 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f8d1ba6-734b-4c0f-a987-d2559ac98be9 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:26.128906 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28ea2201-bc54-42e8-a31f-0ec9bc3d1acf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:26.129659 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ea2201-bc54-42e8-a31f-0ec9bc3d1acf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:26.130125 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ea2201-bc54-42e8-a31f-0ec9bc3d1acf admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:26.130537 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ea2201-bc54-42e8-a31f-0ec9bc3d1acf admin admin] PATH_INFO: `/v3/projects/c03a06d2e93a42be9b2279ac67b47dce` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:26.134169 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28ea2201-bc54-42e8-a31f-0ec9bc3d1acf admin admin] RBAC: Authorizing `identity:delete_project(project_id=c03a06d2e93a42be9b2279ac67b47dce)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:26.137368 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28ea2201-bc54-42e8-a31f-0ec9bc3d1acf admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:26.141472 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ee86ba9-2630-4049-80ff-0c4e6d1e2276 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:26.141472 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee86ba9-2630-4049-80ff-0c4e6d1e2276 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:26.141732 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee86ba9-2630-4049-80ff-0c4e6d1e2276 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:26.141978 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee86ba9-2630-4049-80ff-0c4e6d1e2276 admin admin] PATH_INFO: `/v3/users/fbe1f133cdc048ac804b402a7164ccfa` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:26.143558 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ee86ba9-2630-4049-80ff-0c4e6d1e2276 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fbe1f133cdc048ac804b402a7164ccfa)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:26.148026 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ee86ba9-2630-4049-80ff-0c4e6d1e2276 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:26.223531 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-28ea2201-bc54-42e8-a31f-0ec9bc3d1acf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c03a06d2e93a42be9b2279ac67b47dce', 'request_id': 'req-28ea2201-bc54-42e8-a31f-0ec9bc3d1acf'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:26.248534 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-28ea2201-bc54-42e8-a31f-0ec9bc3d1acf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c03a06d2e93a42be9b2279ac67b47dce', 'request_id': 'req-28ea2201-bc54-42e8-a31f-0ec9bc3d1acf'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:26.254107 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-8ee86ba9-2630-4049-80ff-0c4e6d1e2276 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fbe1f133cdc048ac804b402a7164ccfa'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:26.265943 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-28ea2201-bc54-42e8-a31f-0ec9bc3d1acf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c03a06d2e93a42be9b2279ac67b47dce', 'request_id': 'req-28ea2201-bc54-42e8-a31f-0ec9bc3d1acf'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:26.267554 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-8ee86ba9-2630-4049-80ff-0c4e6d1e2276 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fbe1f133cdc048ac804b402a7164ccfa'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:26.286433 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-8ee86ba9-2630-4049-80ff-0c4e6d1e2276 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fbe1f133cdc048ac804b402a7164ccfa'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:26.286794 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-28ea2201-bc54-42e8-a31f-0ec9bc3d1acf admin admin] The token cache is being invalidate because project c03a06d2e93a42be9b2279ac67b47dce was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81562) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:01:26.287282 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-28ea2201-bc54-42e8-a31f-0ec9bc3d1acf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:26.288141 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-28ea2201-bc54-42e8-a31f-0ec9bc3d1acf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:26.294809 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 844/1686] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:01:25 2022] DELETE /identity/v3/projects/c03a06d2e93a42be9b2279ac67b47dce => generated 0 bytes in 343 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:01:26.297714 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-8ee86ba9-2630-4049-80ff-0c4e6d1e2276 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fbe1f133cdc048ac804b402a7164ccfa'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:26.298268 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83669dc8-46a9-4c06-b6ef-c19f030ca64e None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:26.298779 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83669dc8-46a9-4c06-b6ef-c19f030ca64e None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:26.298872 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83669dc8-46a9-4c06-b6ef-c19f030ca64e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:26.299491 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-8ee86ba9-2630-4049-80ff-0c4e6d1e2276 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fbe1f133cdc048ac804b402a7164ccfa'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:26.304783 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:26.310929 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 843/1687] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:01:25 2022] DELETE /identity/v3/users/fbe1f133cdc048ac804b402a7164ccfa => generated 0 bytes in 324 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:26.332496 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-83669dc8-46a9-4c06-b6ef-c19f030ca64e None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:26.333888 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-83669dc8-46a9-4c06-b6ef-c19f030ca64e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:26.414387 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 845/1688] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:26.422180 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e3c9d5-5bc3-42c2-ac16-e007bbc746b4 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:26.423162 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e3c9d5-5bc3-42c2-ac16-e007bbc746b4 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:26.423162 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e3c9d5-5bc3-42c2-ac16-e007bbc746b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:26.444610 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-d2e3c9d5-5bc3-42c2-ac16-e007bbc746b4 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:26.445829 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-d2e3c9d5-5bc3-42c2-ac16-e007bbc746b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:26.453772 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 844/1689] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:26.458055 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83669dc8-46a9-4c06-b6ef-c19f030ca64e None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:26.470209 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83669dc8-46a9-4c06-b6ef-c19f030ca64e None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:26.475049 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b06c1194-ee36-4cb0-8dfa-199e1550e5f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:26.475706 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b06c1194-ee36-4cb0-8dfa-199e1550e5f8 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:26.476567 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b06c1194-ee36-4cb0-8dfa-199e1550e5f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:26.476862 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b06c1194-ee36-4cb0-8dfa-199e1550e5f8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:26.477501 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b06c1194-ee36-4cb0-8dfa-199e1550e5f8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 09:01:26.478202 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b06c1194-ee36-4cb0-8dfa-199e1550e5f8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:26.479435 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b06c1194-ee36-4cb0-8dfa-199e1550e5f8 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:26.492546 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 846/1690] 10.222.0.36 () {58 vars in 1153 bytes} [Thu Oct 20 09:01:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 09:01:26.497209 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2e3c9d5-5bc3-42c2-ac16-e007bbc746b4 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:26.507702 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2e3c9d5-5bc3-42c2-ac16-e007bbc746b4 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:26.516619 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb71658-76b4-4341-b9ad-c5f3e055e91a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:26.517527 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb71658-76b4-4341-b9ad-c5f3e055e91a admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:26.517640 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb71658-76b4-4341-b9ad-c5f3e055e91a admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:26.517872 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb71658-76b4-4341-b9ad-c5f3e055e91a admin admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:26.519638 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebb71658-76b4-4341-b9ad-c5f3e055e91a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:26.521677 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebb71658-76b4-4341-b9ad-c5f3e055e91a admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:26.553894 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 845/1691] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:01:26 2022] POST /identity/v3/projects => generated 373 bytes in 57 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:01:26.558650 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b06c1194-ee36-4cb0-8dfa-199e1550e5f8 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:26.571756 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b06c1194-ee36-4cb0-8dfa-199e1550e5f8 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:26.593091 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:26.642287 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0a42906-e19d-472e-9139-2c722a554a46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:26.643125 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0a42906-e19d-472e-9139-2c722a554a46 admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:26.643674 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0a42906-e19d-472e-9139-2c722a554a46 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:26.644163 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0a42906-e19d-472e-9139-2c722a554a46 admin admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:26.645061 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0a42906-e19d-472e-9139-2c722a554a46 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:26.648800 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0a42906-e19d-472e-9139-2c722a554a46 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:26.674429 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 847/1692] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:01:26 2022] POST /identity/v3/users => generated 287 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 20 09:01:26.678567 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb71658-76b4-4341-b9ad-c5f3e055e91a admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:26.692850 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb71658-76b4-4341-b9ad-c5f3e055e91a admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:26.699546 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9179b7da-c490-4147-b49a-363a308fa70b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:26.700273 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9179b7da-c490-4147-b49a-363a308fa70b admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:26.700620 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9179b7da-c490-4147-b49a-363a308fa70b admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:26.700959 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9179b7da-c490-4147-b49a-363a308fa70b admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:26.701662 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9179b7da-c490-4147-b49a-363a308fa70b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:26.703415 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9179b7da-c490-4147-b49a-363a308fa70b admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:26.718992 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 846/1693] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:01:26 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 20 09:01:26.720771 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0a42906-e19d-472e-9139-2c722a554a46 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:26.733610 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0a42906-e19d-472e-9139-2c722a554a46 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:26.739129 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e288b1ee-0a5c-46bb-80ba-c2ba5da0d8be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:26.739804 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e288b1ee-0a5c-46bb-80ba-c2ba5da0d8be admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:26.740295 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e288b1ee-0a5c-46bb-80ba-c2ba5da0d8be admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:26.740637 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e288b1ee-0a5c-46bb-80ba-c2ba5da0d8be admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:26.741292 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e288b1ee-0a5c-46bb-80ba-c2ba5da0d8be admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:26.742965 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e288b1ee-0a5c-46bb-80ba-c2ba5da0d8be admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:26.756184 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 848/1694] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:01:26 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:01:26.761642 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9179b7da-c490-4147-b49a-363a308fa70b admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:26.773210 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e288b1ee-0a5c-46bb-80ba-c2ba5da0d8be admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:26.775130 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-e288b1ee-0a5c-46bb-80ba-c2ba5da0d8be admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:26.776189 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9179b7da-c490-4147-b49a-363a308fa70b admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:26.784744 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dcd63dc-6569-4c42-a98b-cf923726f535 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:26.785422 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dcd63dc-6569-4c42-a98b-cf923726f535 admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:26.785697 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dcd63dc-6569-4c42-a98b-cf923726f535 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:26.786022 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dcd63dc-6569-4c42-a98b-cf923726f535 admin admin] PATH_INFO: `/v3/projects/1f24684045924d9bbd661bd8a74f751c/users/9c7a132459ba43ed87d9c429e2f59940/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:26.793543 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e288b1ee-0a5c-46bb-80ba-c2ba5da0d8be admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:26.799415 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-997bf484-f28b-481a-a90f-20efa1a9c003 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:26.800241 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-997bf484-f28b-481a-a90f-20efa1a9c003 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:26.800687 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-997bf484-f28b-481a-a90f-20efa1a9c003 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:26.801103 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-997bf484-f28b-481a-a90f-20efa1a9c003 admin admin] PATH_INFO: `/v3/projects/1776e1f445f548b7a54decdabe1c469d` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:26.802897 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-997bf484-f28b-481a-a90f-20efa1a9c003 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1776e1f445f548b7a54decdabe1c469d)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:26.803166 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dcd63dc-6569-4c42-a98b-cf923726f535 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1f24684045924d9bbd661bd8a74f751c, user_id=9c7a132459ba43ed87d9c429e2f59940, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:26.805802 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dcd63dc-6569-4c42-a98b-cf923726f535 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:26.806253 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-997bf484-f28b-481a-a90f-20efa1a9c003 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:26.824051 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 847/1695] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:01:26 2022] PUT /identity/v3/projects/1f24684045924d9bbd661bd8a74f751c/users/9c7a132459ba43ed87d9c429e2f59940/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:26.829050 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2bd831-ac1c-4758-8b02-7cdb0ab05cd7 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:26.829337 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2bd831-ac1c-4758-8b02-7cdb0ab05cd7 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:26.829679 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2bd831-ac1c-4758-8b02-7cdb0ab05cd7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:26.862495 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-0e2bd831-ac1c-4758-8b02-7cdb0ab05cd7 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:26.863906 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-0e2bd831-ac1c-4758-8b02-7cdb0ab05cd7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:26.883685 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-997bf484-f28b-481a-a90f-20efa1a9c003 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1776e1f445f548b7a54decdabe1c469d', 'request_id': 'req-997bf484-f28b-481a-a90f-20efa1a9c003'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:26.900089 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-997bf484-f28b-481a-a90f-20efa1a9c003 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1776e1f445f548b7a54decdabe1c469d', 'request_id': 'req-997bf484-f28b-481a-a90f-20efa1a9c003'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:26.919091 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-997bf484-f28b-481a-a90f-20efa1a9c003 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1776e1f445f548b7a54decdabe1c469d', 'request_id': 'req-997bf484-f28b-481a-a90f-20efa1a9c003'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:26.960121 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-997bf484-f28b-481a-a90f-20efa1a9c003 admin admin] The token cache is being invalidate because project 1776e1f445f548b7a54decdabe1c469d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81562) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:01:26.961226 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-997bf484-f28b-481a-a90f-20efa1a9c003 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:26.963600 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-997bf484-f28b-481a-a90f-20efa1a9c003 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:26.968573 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 849/1696] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:01:26 2022] DELETE /identity/v3/projects/1776e1f445f548b7a54decdabe1c469d => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:01:26.969831 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-997bf484-f28b-481a-a90f-20efa1a9c003 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:26.971684 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-997bf484-f28b-481a-a90f-20efa1a9c003 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:26.996006 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 848/1697] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 20 09:01:27.003376 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e2bd831-ac1c-4758-8b02-7cdb0ab05cd7 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:27.004920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-0e2bd831-ac1c-4758-8b02-7cdb0ab05cd7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:27.063710 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-997bf484-f28b-481a-a90f-20efa1a9c003 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:27.071449 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-486bb1cd-5334-47ea-af81-70b29c9d269c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:27.073470 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-486bb1cd-5334-47ea-af81-70b29c9d269c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:27.073965 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-486bb1cd-5334-47ea-af81-70b29c9d269c admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:27.074657 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-486bb1cd-5334-47ea-af81-70b29c9d269c admin admin] PATH_INFO: `/v3/users/f91335f09b5141d9aa84abb0bacb2215` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:27.077174 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-486bb1cd-5334-47ea-af81-70b29c9d269c admin admin] RBAC: Authorizing `identity:delete_user(user_id=f91335f09b5141d9aa84abb0bacb2215)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:27.086342 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-486bb1cd-5334-47ea-af81-70b29c9d269c admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:27.126294 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e2bd831-ac1c-4758-8b02-7cdb0ab05cd7 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:27.138400 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77266ca0-e624-4565-a1cb-a9da0ec3da04 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:27.139153 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77266ca0-e624-4565-a1cb-a9da0ec3da04 service neutron] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:27.140074 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77266ca0-e624-4565-a1cb-a9da0ec3da04 service neutron] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:27.142119 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77266ca0-e624-4565-a1cb-a9da0ec3da04 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:27.160437 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77266ca0-e624-4565-a1cb-a9da0ec3da04 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:27.162037 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77266ca0-e624-4565-a1cb-a9da0ec3da04 service neutron] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:27.163871 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=85,keystone:INSERT=4,keystone:DELETE=11 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:27.184018 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 849/1698] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:01:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 20 09:01:27.204500 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-486bb1cd-5334-47ea-af81-70b29c9d269c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f91335f09b5141d9aa84abb0bacb2215'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:27.223485 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-486bb1cd-5334-47ea-af81-70b29c9d269c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f91335f09b5141d9aa84abb0bacb2215'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:27.264829 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-486bb1cd-5334-47ea-af81-70b29c9d269c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f91335f09b5141d9aa84abb0bacb2215'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:27.273132 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-486bb1cd-5334-47ea-af81-70b29c9d269c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f91335f09b5141d9aa84abb0bacb2215'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:27.274113 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-486bb1cd-5334-47ea-af81-70b29c9d269c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f91335f09b5141d9aa84abb0bacb2215'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:27.284004 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 850/1699] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:01:26 2022] DELETE /identity/v3/users/f91335f09b5141d9aa84abb0bacb2215 => generated 0 bytes in 315 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:01:28.017536 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77266ca0-e624-4565-a1cb-a9da0ec3da04 service neutron] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:28.030114 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77266ca0-e624-4565-a1cb-a9da0ec3da04 service neutron] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:28.078152 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:28.104871 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0578792-b866-447c-8fb9-fe161552bb24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:28.105741 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0578792-b866-447c-8fb9-fe161552bb24 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:28.106170 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0578792-b866-447c-8fb9-fe161552bb24 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:28.106641 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0578792-b866-447c-8fb9-fe161552bb24 admin admin] PATH_INFO: `/v3/projects/09b8131fb64e44599bc99a9fbe6f22d1` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:28.109709 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0578792-b866-447c-8fb9-fe161552bb24 admin admin] RBAC: Authorizing `identity:delete_project(project_id=09b8131fb64e44599bc99a9fbe6f22d1)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:28.111927 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0578792-b866-447c-8fb9-fe161552bb24 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:28.162875 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09176dc3-520f-4fe6-9305-71986e6a27b0 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:28.162875 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09176dc3-520f-4fe6-9305-71986e6a27b0 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:28.163694 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09176dc3-520f-4fe6-9305-71986e6a27b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:28.176376 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-c0578792-b866-447c-8fb9-fe161552bb24 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '09b8131fb64e44599bc99a9fbe6f22d1', 'request_id': 'req-c0578792-b866-447c-8fb9-fe161552bb24'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:28.184465 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-c0578792-b866-447c-8fb9-fe161552bb24 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '09b8131fb64e44599bc99a9fbe6f22d1', 'request_id': 'req-c0578792-b866-447c-8fb9-fe161552bb24'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:28.195408 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-c0578792-b866-447c-8fb9-fe161552bb24 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '09b8131fb64e44599bc99a9fbe6f22d1', 'request_id': 'req-c0578792-b866-447c-8fb9-fe161552bb24'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:28.205563 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-09176dc3-520f-4fe6-9305-71986e6a27b0 None None] MFA Rules not processed for user `9c7a132459ba43ed87d9c429e2f59940`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:28.207464 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-09176dc3-520f-4fe6-9305-71986e6a27b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:28.211811 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-c0578792-b866-447c-8fb9-fe161552bb24 admin admin] The token cache is being invalidate because project 09b8131fb64e44599bc99a9fbe6f22d1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81561) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:01:28.212101 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-c0578792-b866-447c-8fb9-fe161552bb24 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:28.212734 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-c0578792-b866-447c-8fb9-fe161552bb24 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:28.217925 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 850/1700] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:01:28 2022] DELETE /identity/v3/projects/09b8131fb64e44599bc99a9fbe6f22d1 => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:28.219744 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e407584-b170-47b1-9986-8273b39682fd None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:28.220416 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e407584-b170-47b1-9986-8273b39682fd None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:28.220686 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e407584-b170-47b1-9986-8273b39682fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:28.271859 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-4e407584-b170-47b1-9986-8273b39682fd None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:28.273137 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-4e407584-b170-47b1-9986-8273b39682fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:28.372507 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 851/1701] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:28 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 211 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:28.373609 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09176dc3-520f-4fe6-9305-71986e6a27b0 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:28.379528 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-09176dc3-520f-4fe6-9305-71986e6a27b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:28.435628 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 851/1702] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 217 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:28.443433 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-800b20a9-268a-44b8-baa3-7467276a4a1e None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:28.443713 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-800b20a9-268a-44b8-baa3-7467276a4a1e None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:28.443987 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-800b20a9-268a-44b8-baa3-7467276a4a1e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:28.467357 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-800b20a9-268a-44b8-baa3-7467276a4a1e None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:28.468997 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-800b20a9-268a-44b8-baa3-7467276a4a1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:28.476919 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 852/1703] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 20 09:01:28.481265 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-800b20a9-268a-44b8-baa3-7467276a4a1e None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:28.492095 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-800b20a9-268a-44b8-baa3-7467276a4a1e None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:28.494380 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09176dc3-520f-4fe6-9305-71986e6a27b0 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:28.496806 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc71d2b7-ecea-4916-944d-3d86578a2d77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:28.497415 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc71d2b7-ecea-4916-944d-3d86578a2d77 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:28.497689 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc71d2b7-ecea-4916-944d-3d86578a2d77 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:28.498289 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc71d2b7-ecea-4916-944d-3d86578a2d77 admin admin] PATH_INFO: `/v3/domains` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:28.498977 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc71d2b7-ecea-4916-944d-3d86578a2d77 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 09:01:28.499276 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc71d2b7-ecea-4916-944d-3d86578a2d77 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:28.501069 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc71d2b7-ecea-4916-944d-3d86578a2d77 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:28.514204 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90ee5e3b-9e33-4b65-aaad-e037cd2d4281 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:28.515526 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 853/1704] 10.222.0.36 () {58 vars in 1153 bytes} [Thu Oct 20 09:01:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 09:01:28.516478 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ee5e3b-9e33-4b65-aaad-e037cd2d4281 admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:28.517680 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ee5e3b-9e33-4b65-aaad-e037cd2d4281 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:28.518271 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ee5e3b-9e33-4b65-aaad-e037cd2d4281 admin admin] PATH_INFO: `/v3/projects` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:28.519537 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90ee5e3b-9e33-4b65-aaad-e037cd2d4281 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:28.520263 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc71d2b7-ecea-4916-944d-3d86578a2d77 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:28.524074 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90ee5e3b-9e33-4b65-aaad-e037cd2d4281 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:28.530685 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc71d2b7-ecea-4916-944d-3d86578a2d77 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:28.535863 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a40d084-4c4a-491e-8527-28b7b0eea8ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:28.536455 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a40d084-4c4a-491e-8527-28b7b0eea8ac admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:28.536730 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a40d084-4c4a-491e-8527-28b7b0eea8ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:28.536996 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a40d084-4c4a-491e-8527-28b7b0eea8ac admin admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:28.537622 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a40d084-4c4a-491e-8527-28b7b0eea8ac admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:28.539574 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a40d084-4c4a-491e-8527-28b7b0eea8ac admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:28.561086 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 852/1705] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:01:28 2022] POST /identity/v3/projects => generated 373 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:01:28.565342 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90ee5e3b-9e33-4b65-aaad-e037cd2d4281 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:28.575779 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 854/1706] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:01:28 2022] POST /identity/v3/projects => generated 371 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:01:28.582349 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a40d084-4c4a-491e-8527-28b7b0eea8ac admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:28.596996 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:DELETE=6,keystone:SELECT=92,keystone:INSERT=2 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:28.624256 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:28.675052 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a40d084-4c4a-491e-8527-28b7b0eea8ac admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:28.679072 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90ee5e3b-9e33-4b65-aaad-e037cd2d4281 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:28.683125 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-524e6a02-1ac0-495e-b573-1f43c9796d87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:28.683746 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-524e6a02-1ac0-495e-b573-1f43c9796d87 admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:28.684022 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-524e6a02-1ac0-495e-b573-1f43c9796d87 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:28.684777 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-524e6a02-1ac0-495e-b573-1f43c9796d87 admin admin] PATH_INFO: `/v3/users` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:28.686618 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-524e6a02-1ac0-495e-b573-1f43c9796d87 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:28.688190 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-524e6a02-1ac0-495e-b573-1f43c9796d87 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:28.689960 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ad8ad8-0495-4e6c-9bce-418fd43150ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:28.690545 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ad8ad8-0495-4e6c-9bce-418fd43150ae admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:28.690824 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ad8ad8-0495-4e6c-9bce-418fd43150ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:28.691092 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ad8ad8-0495-4e6c-9bce-418fd43150ae admin admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:28.691829 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6ad8ad8-0495-4e6c-9bce-418fd43150ae admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:28.693737 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6ad8ad8-0495-4e6c-9bce-418fd43150ae admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:28.722782 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 855/1707] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:01:28 2022] POST /identity/v3/users => generated 286 bytes in 139 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:01:28.724256 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-524e6a02-1ac0-495e-b573-1f43c9796d87 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:28.727404 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 853/1708] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:01:28 2022] POST /identity/v3/users => generated 293 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:01:28.731472 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ad8ad8-0495-4e6c-9bce-418fd43150ae admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:28.734733 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-524e6a02-1ac0-495e-b573-1f43c9796d87 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:28.742061 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ad8ad8-0495-4e6c-9bce-418fd43150ae admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:28.746836 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-809e78b0-9db0-42b0-93f0-b38552ddf41a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:28.747438 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-809e78b0-9db0-42b0-93f0-b38552ddf41a admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:28.747718 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-809e78b0-9db0-42b0-93f0-b38552ddf41a admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:28.747990 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-809e78b0-9db0-42b0-93f0-b38552ddf41a admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:28.748773 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-809e78b0-9db0-42b0-93f0-b38552ddf41a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:28.750379 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-809e78b0-9db0-42b0-93f0-b38552ddf41a admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:28.754296 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa961dba-c406-486b-adbf-aaef8e26798d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:28.754868 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa961dba-c406-486b-adbf-aaef8e26798d admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:28.755139 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa961dba-c406-486b-adbf-aaef8e26798d admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:28.755406 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa961dba-c406-486b-adbf-aaef8e26798d admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:28.756082 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa961dba-c406-486b-adbf-aaef8e26798d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:28.758801 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa961dba-c406-486b-adbf-aaef8e26798d admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:28.762999 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 854/1709] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:01:28 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:01:28.767198 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-809e78b0-9db0-42b0-93f0-b38552ddf41a admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:28.770522 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 856/1710] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:01:28 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 20 09:01:28.774618 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa961dba-c406-486b-adbf-aaef8e26798d admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:28.777494 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-809e78b0-9db0-42b0-93f0-b38552ddf41a admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:28.784990 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa961dba-c406-486b-adbf-aaef8e26798d admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:28.786262 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57682435-cc0c-4505-a82c-359b5a8ce488 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:28.786950 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57682435-cc0c-4505-a82c-359b5a8ce488 admin admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:28.787228 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57682435-cc0c-4505-a82c-359b5a8ce488 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:28.787499 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57682435-cc0c-4505-a82c-359b5a8ce488 admin admin] PATH_INFO: `/v3/projects/5a5c6b9deac74d5f92cbd17bc0ca0cd2/users/b4efa5e581894a6bae36c3c4b56801c0/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:28.790224 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49ec3222-3f9e-4b01-8cbf-c1374ae81f25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:28.790994 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49ec3222-3f9e-4b01-8cbf-c1374ae81f25 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:28.791331 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49ec3222-3f9e-4b01-8cbf-c1374ae81f25 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:28.791686 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49ec3222-3f9e-4b01-8cbf-c1374ae81f25 admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:28.792396 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49ec3222-3f9e-4b01-8cbf-c1374ae81f25 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:28.794071 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49ec3222-3f9e-4b01-8cbf-c1374ae81f25 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:28.808898 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 857/1711] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:01:28 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 20 09:01:28.809956 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49ec3222-3f9e-4b01-8cbf-c1374ae81f25 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:28.815755 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57682435-cc0c-4505-a82c-359b5a8ce488 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5a5c6b9deac74d5f92cbd17bc0ca0cd2, user_id=b4efa5e581894a6bae36c3c4b56801c0, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:28.819398 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57682435-cc0c-4505-a82c-359b5a8ce488 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:28.821880 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49ec3222-3f9e-4b01-8cbf-c1374ae81f25 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:28.826664 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38784130-23f5-4ef5-b94c-62b45604f3b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:28.827313 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38784130-23f5-4ef5-b94c-62b45604f3b8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:28.827639 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38784130-23f5-4ef5-b94c-62b45604f3b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:28.827935 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38784130-23f5-4ef5-b94c-62b45604f3b8 admin admin] PATH_INFO: `/v3/projects/847f0b8423334fc7b505313a10845bcb/users/930afcdb977f4ed2a33b366563a755f1/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:28.845235 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38784130-23f5-4ef5-b94c-62b45604f3b8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=847f0b8423334fc7b505313a10845bcb, user_id=930afcdb977f4ed2a33b366563a755f1, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:28.845719 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 855/1712] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:01:28 2022] PUT /identity/v3/projects/5a5c6b9deac74d5f92cbd17bc0ca0cd2/users/b4efa5e581894a6bae36c3c4b56801c0/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 79 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:28.848528 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38784130-23f5-4ef5-b94c-62b45604f3b8 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:28.850988 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57682435-cc0c-4505-a82c-359b5a8ce488 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:28.866162 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 858/1713] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:01:28 2022] PUT /identity/v3/projects/847f0b8423334fc7b505313a10845bcb/users/930afcdb977f4ed2a33b366563a755f1/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:28.871679 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd896c00-eb01-455a-b89c-8503324af9ac None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:28.872014 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd896c00-eb01-455a-b89c-8503324af9ac None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:28.872292 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd896c00-eb01-455a-b89c-8503324af9ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:28.896873 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-cd896c00-eb01-455a-b89c-8503324af9ac None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:28.898150 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-cd896c00-eb01-455a-b89c-8503324af9ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:28.930910 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57682435-cc0c-4505-a82c-359b5a8ce488 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:28.985298 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 859/1714] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 20 09:01:29.003947 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd896c00-eb01-455a-b89c-8503324af9ac None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:29.005680 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-cd896c00-eb01-455a-b89c-8503324af9ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:29.026662 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f39263f1-9428-4303-a1e8-6e66427280e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:29.027272 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f39263f1-9428-4303-a1e8-6e66427280e9 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:29.027548 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f39263f1-9428-4303-a1e8-6e66427280e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:29.028218 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f39263f1-9428-4303-a1e8-6e66427280e9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:29.028452 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f39263f1-9428-4303-a1e8-6e66427280e9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:29.030035 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f39263f1-9428-4303-a1e8-6e66427280e9 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:29.043591 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 856/1715] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:01:28 2022] GET /identity/v3/roles => generated 1420 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:01:29.044751 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f39263f1-9428-4303-a1e8-6e66427280e9 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:29.057122 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f39263f1-9428-4303-a1e8-6e66427280e9 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:29.062717 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd657230-d6f4-4fe6-a9a3-4995bf30735d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:29.063464 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd657230-d6f4-4fe6-a9a3-4995bf30735d admin admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:29.063818 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd657230-d6f4-4fe6-a9a3-4995bf30735d admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:29.064161 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd657230-d6f4-4fe6-a9a3-4995bf30735d admin admin] PATH_INFO: `/v3/projects/5a5c6b9deac74d5f92cbd17bc0ca0cd2/users/b4efa5e581894a6bae36c3c4b56801c0/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:29.066167 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd657230-d6f4-4fe6-a9a3-4995bf30735d admin admin] RBAC: Authorizing `identity:create_grant(project_id=5a5c6b9deac74d5f92cbd17bc0ca0cd2, user_id=b4efa5e581894a6bae36c3c4b56801c0, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:29.068914 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd657230-d6f4-4fe6-a9a3-4995bf30735d admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:29.069112 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd896c00-eb01-455a-b89c-8503324af9ac None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:29.075430 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-035b83ae-a2ff-4894-bda0-3b452bb1eeea service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:29.076605 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035b83ae-a2ff-4894-bda0-3b452bb1eeea service neutron] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:29.076893 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035b83ae-a2ff-4894-bda0-3b452bb1eeea service neutron] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:29.077160 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035b83ae-a2ff-4894-bda0-3b452bb1eeea service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:29.081290 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 857/1716] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:01:29 2022] PUT /identity/v3/projects/5a5c6b9deac74d5f92cbd17bc0ca0cd2/users/b4efa5e581894a6bae36c3c4b56801c0/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:01:29.090339 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-035b83ae-a2ff-4894-bda0-3b452bb1eeea service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:29.091990 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-035b83ae-a2ff-4894-bda0-3b452bb1eeea service neutron] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:29.103464 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 860/1717] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:01:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 20 09:01:30.102676 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f294752a-a2d2-4f57-8f35-5678fca3a097 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:30.103201 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f294752a-a2d2-4f57-8f35-5678fca3a097 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:30.103201 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f294752a-a2d2-4f57-8f35-5678fca3a097 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:30.116110 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:30.148305 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-f294752a-a2d2-4f57-8f35-5678fca3a097 None None] MFA Rules not processed for user `b4efa5e581894a6bae36c3c4b56801c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:30.149848 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-f294752a-a2d2-4f57-8f35-5678fca3a097 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:30.239273 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 858/1718] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:30 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:30.251700 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-035b83ae-a2ff-4894-bda0-3b452bb1eeea service neutron] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:30.253245 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-035b83ae-a2ff-4894-bda0-3b452bb1eeea service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:30.272661 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:30.327683 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-035b83ae-a2ff-4894-bda0-3b452bb1eeea service neutron] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:30.386109 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3abc8f-632d-496d-9c25-e53be2539b3e None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:30.386693 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3abc8f-632d-496d-9c25-e53be2539b3e None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:30.387105 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3abc8f-632d-496d-9c25-e53be2539b3e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:30.392858 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-320256a7-8f78-419d-81d1-a0cd6e8254bc service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:30.394149 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320256a7-8f78-419d-81d1-a0cd6e8254bc service cinder] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:30.394424 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320256a7-8f78-419d-81d1-a0cd6e8254bc service cinder] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:30.394964 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320256a7-8f78-419d-81d1-a0cd6e8254bc service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:30.421975 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-7c3abc8f-632d-496d-9c25-e53be2539b3e None None] MFA Rules not processed for user `930afcdb977f4ed2a33b366563a755f1`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:30.423340 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-7c3abc8f-632d-496d-9c25-e53be2539b3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:30.462715 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-320256a7-8f78-419d-81d1-a0cd6e8254bc service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:30.464322 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-320256a7-8f78-419d-81d1-a0cd6e8254bc service cinder] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:30.474951 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 861/1719] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:01:30 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 224 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:01:30.498586 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 859/1720] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:30 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 20 09:01:30.549335 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-320256a7-8f78-419d-81d1-a0cd6e8254bc service cinder] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:30.550790 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-320256a7-8f78-419d-81d1-a0cd6e8254bc service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:30.564627 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-320256a7-8f78-419d-81d1-a0cd6e8254bc service cinder] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:30.568463 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ad99707-b8b1-48cd-a520-9f51e2176a5d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:30.569071 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad99707-b8b1-48cd-a520-9f51e2176a5d service cinder] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:30.569431 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad99707-b8b1-48cd-a520-9f51e2176a5d service cinder] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:30.570185 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad99707-b8b1-48cd-a520-9f51e2176a5d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:30.587112 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ad99707-b8b1-48cd-a520-9f51e2176a5d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:30.589254 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ad99707-b8b1-48cd-a520-9f51e2176a5d service cinder] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:30.609196 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 862/1721] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:01:30 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 60 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:01:35.209584 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c3abc8f-632d-496d-9c25-e53be2539b3e None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:35.220790 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c3abc8f-632d-496d-9c25-e53be2539b3e None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:35.224105 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1d2e6fb-43c1-49e4-bc72-f837beed1609 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:35.224826 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d2e6fb-43c1-49e4-bc72-f837beed1609 service cinder] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:35.225205 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d2e6fb-43c1-49e4-bc72-f837beed1609 service cinder] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:35.225610 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d2e6fb-43c1-49e4-bc72-f837beed1609 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:35.227651 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be6569d9-b152-4094-8404-072f3089c722 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:35.228106 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be6569d9-b152-4094-8404-072f3089c722 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:35.228309 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be6569d9-b152-4094-8404-072f3089c722 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:35.235624 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1d2e6fb-43c1-49e4-bc72-f837beed1609 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:35.237119 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1d2e6fb-43c1-49e4-bc72-f837beed1609 service cinder] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:35.247421 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 860/1722] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:01:35 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 20 09:01:35.250877 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-be6569d9-b152-4094-8404-072f3089c722 None None] MFA Rules not processed for user `930afcdb977f4ed2a33b366563a755f1`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:35.252102 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-be6569d9-b152-4094-8404-072f3089c722 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:35.259696 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 863/1723] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:35 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 20 09:01:37.413254 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1d2e6fb-43c1-49e4-bc72-f837beed1609 service cinder] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:37.415984 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-d1d2e6fb-43c1-49e4-bc72-f837beed1609 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:37.486790 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1d2e6fb-43c1-49e4-bc72-f837beed1609 service cinder] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:37.491815 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a9d37f8-bb51-4246-b0cc-a65e2b9774ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:37.493082 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a9d37f8-bb51-4246-b0cc-a65e2b9774ce admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:37.493544 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a9d37f8-bb51-4246-b0cc-a65e2b9774ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:37.494155 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a9d37f8-bb51-4246-b0cc-a65e2b9774ce admin admin] PATH_INFO: `/v3/users/06e79ff08f0b45d1aedad0824bb0c62f` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:37.495549 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a9d37f8-bb51-4246-b0cc-a65e2b9774ce admin admin] RBAC: Authorizing `identity:delete_user(user_id=06e79ff08f0b45d1aedad0824bb0c62f)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:37.497553 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a9d37f8-bb51-4246-b0cc-a65e2b9774ce admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:37.507366 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:37.564291 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-4a9d37f8-bb51-4246-b0cc-a65e2b9774ce admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '06e79ff08f0b45d1aedad0824bb0c62f'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:37.574677 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-4a9d37f8-bb51-4246-b0cc-a65e2b9774ce admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '06e79ff08f0b45d1aedad0824bb0c62f'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:37.587906 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-4a9d37f8-bb51-4246-b0cc-a65e2b9774ce admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '06e79ff08f0b45d1aedad0824bb0c62f'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:37.597141 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-4a9d37f8-bb51-4246-b0cc-a65e2b9774ce admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '06e79ff08f0b45d1aedad0824bb0c62f'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:37.598001 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-4a9d37f8-bb51-4246-b0cc-a65e2b9774ce admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '06e79ff08f0b45d1aedad0824bb0c62f'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:37.605300 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 861/1724] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:01:37 2022] DELETE /identity/v3/users/06e79ff08f0b45d1aedad0824bb0c62f => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:37.609755 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e35fa7d-e744-49bf-bdba-554da9ee9792 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:37.610225 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e35fa7d-e744-49bf-bdba-554da9ee9792 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:37.610378 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e35fa7d-e744-49bf-bdba-554da9ee9792 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:37.632287 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-0e35fa7d-e744-49bf-bdba-554da9ee9792 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:37.633510 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-0e35fa7d-e744-49bf-bdba-554da9ee9792 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:37.692001 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 864/1725] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:37.701814 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a9d37f8-bb51-4246-b0cc-a65e2b9774ce admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:37.703315 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-4a9d37f8-bb51-4246-b0cc-a65e2b9774ce admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:37.760875 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a9d37f8-bb51-4246-b0cc-a65e2b9774ce admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:37.764713 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a453e46-eb15-473a-8f03-cb3cc2a31710 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:37.765489 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a453e46-eb15-473a-8f03-cb3cc2a31710 service neutron] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:37.765875 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a453e46-eb15-473a-8f03-cb3cc2a31710 service neutron] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:37.766098 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a453e46-eb15-473a-8f03-cb3cc2a31710 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:37.780433 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a453e46-eb15-473a-8f03-cb3cc2a31710 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:37.784069 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a453e46-eb15-473a-8f03-cb3cc2a31710 service neutron] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:37.797371 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 862/1726] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:01:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:01:38.377537 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e35fa7d-e744-49bf-bdba-554da9ee9792 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:38.379043 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-0e35fa7d-e744-49bf-bdba-554da9ee9792 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:38.391103 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:38.396009 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e35fa7d-e744-49bf-bdba-554da9ee9792 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:38.401939 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9973526e-686e-4b8e-9055-48b041cd593f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:38.402605 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9973526e-686e-4b8e-9055-48b041cd593f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:38.402776 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9973526e-686e-4b8e-9055-48b041cd593f admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:38.403039 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9973526e-686e-4b8e-9055-48b041cd593f admin admin] PATH_INFO: `/v3/projects/cc24996ae3c240a0a8412a3b526d0f4a` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:38.404151 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9973526e-686e-4b8e-9055-48b041cd593f admin admin] RBAC: Authorizing `identity:delete_project(project_id=cc24996ae3c240a0a8412a3b526d0f4a)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:38.406047 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9973526e-686e-4b8e-9055-48b041cd593f admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:38.465982 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-9973526e-686e-4b8e-9055-48b041cd593f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cc24996ae3c240a0a8412a3b526d0f4a', 'request_id': 'req-9973526e-686e-4b8e-9055-48b041cd593f'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:38.473229 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-9973526e-686e-4b8e-9055-48b041cd593f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cc24996ae3c240a0a8412a3b526d0f4a', 'request_id': 'req-9973526e-686e-4b8e-9055-48b041cd593f'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:38.480299 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-9973526e-686e-4b8e-9055-48b041cd593f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cc24996ae3c240a0a8412a3b526d0f4a', 'request_id': 'req-9973526e-686e-4b8e-9055-48b041cd593f'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:38.497671 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-9973526e-686e-4b8e-9055-48b041cd593f admin admin] The token cache is being invalidate because project cc24996ae3c240a0a8412a3b526d0f4a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81561) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:01:38.497971 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-9973526e-686e-4b8e-9055-48b041cd593f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:38.498676 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-9973526e-686e-4b8e-9055-48b041cd593f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:38.500657 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 865/1727] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:01:38 2022] DELETE /identity/v3/projects/cc24996ae3c240a0a8412a3b526d0f4a => generated 0 bytes in 124 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:38.505058 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a453e46-eb15-473a-8f03-cb3cc2a31710 service neutron] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:38.516138 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a453e46-eb15-473a-8f03-cb3cc2a31710 service neutron] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:38.577634 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d33ace49-8aff-4370-a7b6-ad2499f0156a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:38.578272 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d33ace49-8aff-4370-a7b6-ad2499f0156a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:38.578689 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d33ace49-8aff-4370-a7b6-ad2499f0156a admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:38.578974 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d33ace49-8aff-4370-a7b6-ad2499f0156a admin admin] PATH_INFO: `/v3/users/c0087161752b4c789e34d0c40b420483` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:38.580095 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d33ace49-8aff-4370-a7b6-ad2499f0156a admin admin] RBAC: Authorizing `identity:delete_user(user_id=c0087161752b4c789e34d0c40b420483)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:38.582030 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d33ace49-8aff-4370-a7b6-ad2499f0156a admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:38.611950 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=84,keystone:DELETE=15,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:38.631537 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-d33ace49-8aff-4370-a7b6-ad2499f0156a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c0087161752b4c789e34d0c40b420483'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:38.640138 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-d33ace49-8aff-4370-a7b6-ad2499f0156a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c0087161752b4c789e34d0c40b420483'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:38.660040 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-d33ace49-8aff-4370-a7b6-ad2499f0156a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c0087161752b4c789e34d0c40b420483'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:38.672786 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-d33ace49-8aff-4370-a7b6-ad2499f0156a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c0087161752b4c789e34d0c40b420483'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:38.675334 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-d33ace49-8aff-4370-a7b6-ad2499f0156a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c0087161752b4c789e34d0c40b420483'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:38.687918 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 863/1728] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:01:38 2022] DELETE /identity/v3/users/c0087161752b4c789e34d0c40b420483 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:39.060265 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9973526e-686e-4b8e-9055-48b041cd593f admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:39.062073 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-9973526e-686e-4b8e-9055-48b041cd593f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:39.075334 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9973526e-686e-4b8e-9055-48b041cd593f admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:39.144467 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7381295b-6ede-4251-aebb-ea82766e3b67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:39.145542 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7381295b-6ede-4251-aebb-ea82766e3b67 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:39.145799 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7381295b-6ede-4251-aebb-ea82766e3b67 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:39.146268 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7381295b-6ede-4251-aebb-ea82766e3b67 admin admin] PATH_INFO: `/v3/projects/39cc92ab78a142629dff196154f19c5c` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:39.148175 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7381295b-6ede-4251-aebb-ea82766e3b67 admin admin] RBAC: Authorizing `identity:delete_project(project_id=39cc92ab78a142629dff196154f19c5c)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:39.150025 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7381295b-6ede-4251-aebb-ea82766e3b67 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:39.204010 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-7381295b-6ede-4251-aebb-ea82766e3b67 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '39cc92ab78a142629dff196154f19c5c', 'request_id': 'req-7381295b-6ede-4251-aebb-ea82766e3b67'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:39.211374 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-7381295b-6ede-4251-aebb-ea82766e3b67 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '39cc92ab78a142629dff196154f19c5c', 'request_id': 'req-7381295b-6ede-4251-aebb-ea82766e3b67'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:39.224389 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-7381295b-6ede-4251-aebb-ea82766e3b67 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '39cc92ab78a142629dff196154f19c5c', 'request_id': 'req-7381295b-6ede-4251-aebb-ea82766e3b67'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:39.237349 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-7381295b-6ede-4251-aebb-ea82766e3b67 admin admin] The token cache is being invalidate because project 39cc92ab78a142629dff196154f19c5c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81561) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:01:39.237637 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-7381295b-6ede-4251-aebb-ea82766e3b67 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:39.238278 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-7381295b-6ede-4251-aebb-ea82766e3b67 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:39.239786 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 866/1729] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:01:39 2022] DELETE /identity/v3/projects/39cc92ab78a142629dff196154f19c5c => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:39.244784 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7406e4c4-38ec-4dab-9e89-a025979e7075 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:39.245073 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7406e4c4-38ec-4dab-9e89-a025979e7075 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:39.245348 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7406e4c4-38ec-4dab-9e89-a025979e7075 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:39.266007 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-7406e4c4-38ec-4dab-9e89-a025979e7075 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:39.267291 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-7406e4c4-38ec-4dab-9e89-a025979e7075 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:39.330424 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 864/1730] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:39.338563 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b7b05b5-e6eb-41f8-a828-5430f04de7a4 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:39.338853 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b7b05b5-e6eb-41f8-a828-5430f04de7a4 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:39.339125 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b7b05b5-e6eb-41f8-a828-5430f04de7a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:39.360762 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-6b7b05b5-e6eb-41f8-a828-5430f04de7a4 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:39.362059 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-6b7b05b5-e6eb-41f8-a828-5430f04de7a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:39.370990 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 867/1731] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:39.375011 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7406e4c4-38ec-4dab-9e89-a025979e7075 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:39.385289 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7406e4c4-38ec-4dab-9e89-a025979e7075 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:39.389846 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10956264-a49e-4fd7-b4c2-23537fb9b65b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:39.390560 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10956264-a49e-4fd7-b4c2-23537fb9b65b admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:39.390900 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10956264-a49e-4fd7-b4c2-23537fb9b65b admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:39.391237 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10956264-a49e-4fd7-b4c2-23537fb9b65b admin admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:39.392012 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10956264-a49e-4fd7-b4c2-23537fb9b65b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 09:01:39.392442 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10956264-a49e-4fd7-b4c2-23537fb9b65b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:39.394115 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10956264-a49e-4fd7-b4c2-23537fb9b65b admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:39.406811 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 865/1732] 10.222.0.36 () {58 vars in 1153 bytes} [Thu Oct 20 09:01:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 09:01:39.411075 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b7b05b5-e6eb-41f8-a828-5430f04de7a4 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:39.420870 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b7b05b5-e6eb-41f8-a828-5430f04de7a4 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:39.425523 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27eb7400-ba75-451f-bc5a-bc2f6cba3c71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:39.426184 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27eb7400-ba75-451f-bc5a-bc2f6cba3c71 admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:39.426608 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27eb7400-ba75-451f-bc5a-bc2f6cba3c71 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:39.426938 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27eb7400-ba75-451f-bc5a-bc2f6cba3c71 admin admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:39.427659 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27eb7400-ba75-451f-bc5a-bc2f6cba3c71 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:39.430079 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27eb7400-ba75-451f-bc5a-bc2f6cba3c71 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:39.440039 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:39.463011 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 868/1733] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:01:39 2022] POST /identity/v3/projects => generated 381 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:01:39.464780 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10956264-a49e-4fd7-b4c2-23537fb9b65b admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:39.475535 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10956264-a49e-4fd7-b4c2-23537fb9b65b admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:39.534945 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aa5e4d7-cf08-497e-a279-b6eb15760e4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:39.535608 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa5e4d7-cf08-497e-a279-b6eb15760e4e admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:39.535873 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa5e4d7-cf08-497e-a279-b6eb15760e4e admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:39.536057 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa5e4d7-cf08-497e-a279-b6eb15760e4e admin admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:39.536683 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aa5e4d7-cf08-497e-a279-b6eb15760e4e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:39.538584 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aa5e4d7-cf08-497e-a279-b6eb15760e4e admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:39.562504 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 866/1734] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:01:39 2022] POST /identity/v3/users => generated 291 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 20 09:01:39.564013 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27eb7400-ba75-451f-bc5a-bc2f6cba3c71 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:39.574970 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27eb7400-ba75-451f-bc5a-bc2f6cba3c71 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:39.579549 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6109c035-57c1-476e-a1b4-e9657595e86e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:39.580142 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6109c035-57c1-476e-a1b4-e9657595e86e admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:39.580423 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6109c035-57c1-476e-a1b4-e9657595e86e admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:39.580695 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6109c035-57c1-476e-a1b4-e9657595e86e admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:39.581329 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6109c035-57c1-476e-a1b4-e9657595e86e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:39.583001 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6109c035-57c1-476e-a1b4-e9657595e86e admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:39.593118 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 869/1735] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:01:39 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:01:39.596898 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aa5e4d7-cf08-497e-a279-b6eb15760e4e admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:39.607363 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aa5e4d7-cf08-497e-a279-b6eb15760e4e admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:39.611967 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3465a6ba-2c85-40a6-aa3b-8958d7f8794d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:39.612550 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3465a6ba-2c85-40a6-aa3b-8958d7f8794d admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:39.612826 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3465a6ba-2c85-40a6-aa3b-8958d7f8794d admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:39.613095 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3465a6ba-2c85-40a6-aa3b-8958d7f8794d admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:39.613728 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3465a6ba-2c85-40a6-aa3b-8958d7f8794d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:39.615339 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3465a6ba-2c85-40a6-aa3b-8958d7f8794d admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:39.626304 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 867/1736] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:01:39 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:01:39.630304 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6109c035-57c1-476e-a1b4-e9657595e86e admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:39.640190 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6109c035-57c1-476e-a1b4-e9657595e86e admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:39.644696 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e59ec013-9542-44bd-8ba1-083ee5add7ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:39.645330 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e59ec013-9542-44bd-8ba1-083ee5add7ad admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:39.645606 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e59ec013-9542-44bd-8ba1-083ee5add7ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:39.645878 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e59ec013-9542-44bd-8ba1-083ee5add7ad admin admin] PATH_INFO: `/v3/projects/7dabb5b09aea48ea8840befb8d767139/users/baeda7b2fefd446cb9326bed72f9d878/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:39.660204 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e59ec013-9542-44bd-8ba1-083ee5add7ad admin admin] RBAC: Authorizing `identity:create_grant(project_id=7dabb5b09aea48ea8840befb8d767139, user_id=baeda7b2fefd446cb9326bed72f9d878, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:39.662943 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e59ec013-9542-44bd-8ba1-083ee5add7ad admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:39.675961 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 870/1737] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:01:39 2022] PUT /identity/v3/projects/7dabb5b09aea48ea8840befb8d767139/users/baeda7b2fefd446cb9326bed72f9d878/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:39.687726 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd78d7d-3ebc-4aa7-aed2-c6930567b168 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:39.688007 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd78d7d-3ebc-4aa7-aed2-c6930567b168 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:39.688279 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd78d7d-3ebc-4aa7-aed2-c6930567b168 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:39.708647 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=91,keystone:DELETE=5,keystone:INSERT=4 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:39.717397 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-2fd78d7d-3ebc-4aa7-aed2-c6930567b168 None None] MFA Rules not processed for user `baeda7b2fefd446cb9326bed72f9d878`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:39.719151 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-2fd78d7d-3ebc-4aa7-aed2-c6930567b168 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:39.781617 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 868/1738] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:39 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:39.786169 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e59ec013-9542-44bd-8ba1-083ee5add7ad admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:39.847698 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e59ec013-9542-44bd-8ba1-083ee5add7ad admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:39.852786 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc4246b5-0c30-4b24-9490-03e13b2cfcae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:39.853405 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc4246b5-0c30-4b24-9490-03e13b2cfcae admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:39.853684 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc4246b5-0c30-4b24-9490-03e13b2cfcae admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:39.853957 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc4246b5-0c30-4b24-9490-03e13b2cfcae admin admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:39.854613 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc4246b5-0c30-4b24-9490-03e13b2cfcae admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:39.856528 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc4246b5-0c30-4b24-9490-03e13b2cfcae admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:39.884125 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 871/1739] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:01:39 2022] POST /identity/v3/projects => generated 381 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:01:39.891880 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fd78d7d-3ebc-4aa7-aed2-c6930567b168 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:39.909207 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fd78d7d-3ebc-4aa7-aed2-c6930567b168 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:39.968421 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eb1d578-dba5-44bc-9db5-d773dd1f3108 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:39.969278 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eb1d578-dba5-44bc-9db5-d773dd1f3108 admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:39.969708 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eb1d578-dba5-44bc-9db5-d773dd1f3108 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:39.970459 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eb1d578-dba5-44bc-9db5-d773dd1f3108 admin admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:39.970664 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eb1d578-dba5-44bc-9db5-d773dd1f3108 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:39.972545 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eb1d578-dba5-44bc-9db5-d773dd1f3108 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:40.006773 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 869/1740] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:01:39 2022] POST /identity/v3/users => generated 297 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:01:40.010909 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc4246b5-0c30-4b24-9490-03e13b2cfcae admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:40.022491 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc4246b5-0c30-4b24-9490-03e13b2cfcae admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:40.029416 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4ea5515-e81a-49d1-a1f2-3e3ef3cf70ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:40.030941 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ea5515-e81a-49d1-a1f2-3e3ef3cf70ee admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:40.032496 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ea5515-e81a-49d1-a1f2-3e3ef3cf70ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:40.032779 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ea5515-e81a-49d1-a1f2-3e3ef3cf70ee admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:40.033436 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4ea5515-e81a-49d1-a1f2-3e3ef3cf70ee admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:40.035094 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4ea5515-e81a-49d1-a1f2-3e3ef3cf70ee admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:40.048252 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 872/1741] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:01:40 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:01:40.052393 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eb1d578-dba5-44bc-9db5-d773dd1f3108 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:40.063729 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eb1d578-dba5-44bc-9db5-d773dd1f3108 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:40.070772 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-070ce43e-b7fd-4d2d-b7a5-9247bcea71d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:40.071448 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070ce43e-b7fd-4d2d-b7a5-9247bcea71d0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:40.071727 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070ce43e-b7fd-4d2d-b7a5-9247bcea71d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:40.071996 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070ce43e-b7fd-4d2d-b7a5-9247bcea71d0 admin admin] PATH_INFO: `/v3/projects/0d53076c86a64a82a403402e1bb4f64b/users/c0984bce109b49dd9c37f50c2b3f924d/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:40.087672 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-070ce43e-b7fd-4d2d-b7a5-9247bcea71d0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0d53076c86a64a82a403402e1bb4f64b, user_id=c0984bce109b49dd9c37f50c2b3f924d, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:40.090597 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-070ce43e-b7fd-4d2d-b7a5-9247bcea71d0 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:40.104683 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 870/1742] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:01:40 2022] PUT /identity/v3/projects/0d53076c86a64a82a403402e1bb4f64b/users/c0984bce109b49dd9c37f50c2b3f924d/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:01:40.108849 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4ea5515-e81a-49d1-a1f2-3e3ef3cf70ee admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:40.120013 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4ea5515-e81a-49d1-a1f2-3e3ef3cf70ee admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:40.183986 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe7f8463-ff45-41ef-a98b-bf3c8bcbfa1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:40.184816 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe7f8463-ff45-41ef-a98b-bf3c8bcbfa1c admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:40.185092 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe7f8463-ff45-41ef-a98b-bf3c8bcbfa1c admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:40.186279 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe7f8463-ff45-41ef-a98b-bf3c8bcbfa1c admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:40.186279 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe7f8463-ff45-41ef-a98b-bf3c8bcbfa1c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:40.187771 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe7f8463-ff45-41ef-a98b-bf3c8bcbfa1c admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:40.201949 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 873/1743] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:01:40 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:01:40.203302 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-070ce43e-b7fd-4d2d-b7a5-9247bcea71d0 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:40.213781 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-070ce43e-b7fd-4d2d-b7a5-9247bcea71d0 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:40.218625 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ac41fe6-030d-45d4-88f6-bba4dbfcf1dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:40.219262 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ac41fe6-030d-45d4-88f6-bba4dbfcf1dc admin admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:40.219535 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ac41fe6-030d-45d4-88f6-bba4dbfcf1dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:40.219801 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ac41fe6-030d-45d4-88f6-bba4dbfcf1dc admin admin] PATH_INFO: `/v3/projects/0d53076c86a64a82a403402e1bb4f64b/users/c0984bce109b49dd9c37f50c2b3f924d/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:40.221702 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ac41fe6-030d-45d4-88f6-bba4dbfcf1dc admin admin] RBAC: Authorizing `identity:create_grant(project_id=0d53076c86a64a82a403402e1bb4f64b, user_id=c0984bce109b49dd9c37f50c2b3f924d, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:40.224495 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ac41fe6-030d-45d4-88f6-bba4dbfcf1dc admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:40.234490 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 871/1744] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:01:40 2022] PUT /identity/v3/projects/0d53076c86a64a82a403402e1bb4f64b/users/c0984bce109b49dd9c37f50c2b3f924d/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:40.245374 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef70a13d-a12a-4a23-984e-f778652509f0 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:40.245667 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef70a13d-a12a-4a23-984e-f778652509f0 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:40.245941 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef70a13d-a12a-4a23-984e-f778652509f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:40.274770 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-ef70a13d-a12a-4a23-984e-f778652509f0 None None] MFA Rules not processed for user `c0984bce109b49dd9c37f50c2b3f924d`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:40.275957 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-ef70a13d-a12a-4a23-984e-f778652509f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:40.290833 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:40.350582 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 874/1745] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:40 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:40.361738 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ac41fe6-030d-45d4-88f6-bba4dbfcf1dc admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:40.371425 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-2ac41fe6-030d-45d4-88f6-bba4dbfcf1dc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:40.428199 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ac41fe6-030d-45d4-88f6-bba4dbfcf1dc admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:40.431877 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b53d3176-2759-4625-8d13-862b184beaba service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:40.432475 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b53d3176-2759-4625-8d13-862b184beaba service cinder] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:40.432751 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b53d3176-2759-4625-8d13-862b184beaba service cinder] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:40.433021 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b53d3176-2759-4625-8d13-862b184beaba service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:40.439814 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:40.442050 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b53d3176-2759-4625-8d13-862b184beaba service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:40.444046 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b53d3176-2759-4625-8d13-862b184beaba service cinder] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:40.455043 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 872/1746] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:01:40 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:01:40.830680 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef70a13d-a12a-4a23-984e-f778652509f0 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:40.892081 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef70a13d-a12a-4a23-984e-f778652509f0 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:40.896520 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7388969a-9a45-49f6-87d8-1d7f818fa5c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:40.897145 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7388969a-9a45-49f6-87d8-1d7f818fa5c7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:40.897422 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7388969a-9a45-49f6-87d8-1d7f818fa5c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:40.897691 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7388969a-9a45-49f6-87d8-1d7f818fa5c7 admin admin] PATH_INFO: `/v3/users/baeda7b2fefd446cb9326bed72f9d878` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:40.899633 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7388969a-9a45-49f6-87d8-1d7f818fa5c7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=baeda7b2fefd446cb9326bed72f9d878)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:40.901568 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7388969a-9a45-49f6-87d8-1d7f818fa5c7 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:40.951027 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-7388969a-9a45-49f6-87d8-1d7f818fa5c7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'baeda7b2fefd446cb9326bed72f9d878'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:40.958052 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-7388969a-9a45-49f6-87d8-1d7f818fa5c7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'baeda7b2fefd446cb9326bed72f9d878'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:40.970537 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-7388969a-9a45-49f6-87d8-1d7f818fa5c7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'baeda7b2fefd446cb9326bed72f9d878'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:40.978113 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-7388969a-9a45-49f6-87d8-1d7f818fa5c7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'baeda7b2fefd446cb9326bed72f9d878'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:40.978774 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-7388969a-9a45-49f6-87d8-1d7f818fa5c7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'baeda7b2fefd446cb9326bed72f9d878'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:40.984626 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 875/1747] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:01:40 2022] DELETE /identity/v3/users/baeda7b2fefd446cb9326bed72f9d878 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:40.989044 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b79099-9e3d-427e-88e2-cddc2e7dbd0e None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:40.989520 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b79099-9e3d-427e-88e2-cddc2e7dbd0e None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:40.989605 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b79099-9e3d-427e-88e2-cddc2e7dbd0e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:41.008347 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-e6b79099-9e3d-427e-88e2-cddc2e7dbd0e None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:41.009532 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-e6b79099-9e3d-427e-88e2-cddc2e7dbd0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:41.067530 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 873/1748] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:41.077255 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7388969a-9a45-49f6-87d8-1d7f818fa5c7 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:41.078706 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-7388969a-9a45-49f6-87d8-1d7f818fa5c7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:41.112430 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:41.140375 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7388969a-9a45-49f6-87d8-1d7f818fa5c7 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:41.145058 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84fa8162-15b0-4dd6-8c08-730c4b49675d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:41.145785 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84fa8162-15b0-4dd6-8c08-730c4b49675d service neutron] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:41.146156 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84fa8162-15b0-4dd6-8c08-730c4b49675d service neutron] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:41.146522 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84fa8162-15b0-4dd6-8c08-730c4b49675d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:41.157076 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84fa8162-15b0-4dd6-8c08-730c4b49675d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:41.158714 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84fa8162-15b0-4dd6-8c08-730c4b49675d service neutron] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:41.168724 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 876/1749] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:01:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:01:41.479421 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b79099-9e3d-427e-88e2-cddc2e7dbd0e None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:41.480844 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-e6b79099-9e3d-427e-88e2-cddc2e7dbd0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:41.494737 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b79099-9e3d-427e-88e2-cddc2e7dbd0e None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:41.500807 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aced4d7c-e4dc-4dbc-92df-c54659c77c4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:41.501493 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aced4d7c-e4dc-4dbc-92df-c54659c77c4a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:41.501765 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aced4d7c-e4dc-4dbc-92df-c54659c77c4a admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:41.502038 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aced4d7c-e4dc-4dbc-92df-c54659c77c4a admin admin] PATH_INFO: `/v3/projects/7dabb5b09aea48ea8840befb8d767139` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:41.503219 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aced4d7c-e4dc-4dbc-92df-c54659c77c4a admin admin] RBAC: Authorizing `identity:delete_project(project_id=7dabb5b09aea48ea8840befb8d767139)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:41.505185 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aced4d7c-e4dc-4dbc-92df-c54659c77c4a admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:41.552327 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-aced4d7c-e4dc-4dbc-92df-c54659c77c4a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7dabb5b09aea48ea8840befb8d767139', 'request_id': 'req-aced4d7c-e4dc-4dbc-92df-c54659c77c4a'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:41.558930 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-aced4d7c-e4dc-4dbc-92df-c54659c77c4a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7dabb5b09aea48ea8840befb8d767139', 'request_id': 'req-aced4d7c-e4dc-4dbc-92df-c54659c77c4a'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:41.564299 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-aced4d7c-e4dc-4dbc-92df-c54659c77c4a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7dabb5b09aea48ea8840befb8d767139', 'request_id': 'req-aced4d7c-e4dc-4dbc-92df-c54659c77c4a'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:41.577273 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-aced4d7c-e4dc-4dbc-92df-c54659c77c4a admin admin] The token cache is being invalidate because project 7dabb5b09aea48ea8840befb8d767139 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81562) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:01:41.577555 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-aced4d7c-e4dc-4dbc-92df-c54659c77c4a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:41.578184 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-aced4d7c-e4dc-4dbc-92df-c54659c77c4a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:41.579659 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 874/1750] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:01:41 2022] DELETE /identity/v3/projects/7dabb5b09aea48ea8840befb8d767139 => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:41.583739 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84fa8162-15b0-4dd6-8c08-730c4b49675d service neutron] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:41.593687 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84fa8162-15b0-4dd6-8c08-730c4b49675d service neutron] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:41.649331 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2394570-48d5-46f5-a011-59912b862eb5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:41.649911 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2394570-48d5-46f5-a011-59912b862eb5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:41.650195 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2394570-48d5-46f5-a011-59912b862eb5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:41.650498 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2394570-48d5-46f5-a011-59912b862eb5 admin admin] PATH_INFO: `/v3/users/c0984bce109b49dd9c37f50c2b3f924d` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:41.651665 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2394570-48d5-46f5-a011-59912b862eb5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c0984bce109b49dd9c37f50c2b3f924d)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:41.653621 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2394570-48d5-46f5-a011-59912b862eb5 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:41.701916 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-c2394570-48d5-46f5-a011-59912b862eb5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c0984bce109b49dd9c37f50c2b3f924d'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:41.709281 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-c2394570-48d5-46f5-a011-59912b862eb5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c0984bce109b49dd9c37f50c2b3f924d'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:41.722176 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-c2394570-48d5-46f5-a011-59912b862eb5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c0984bce109b49dd9c37f50c2b3f924d'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:41.729220 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-c2394570-48d5-46f5-a011-59912b862eb5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c0984bce109b49dd9c37f50c2b3f924d'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:41.729894 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-c2394570-48d5-46f5-a011-59912b862eb5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c0984bce109b49dd9c37f50c2b3f924d'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:41.735918 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 877/1751] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:01:41 2022] DELETE /identity/v3/users/c0984bce109b49dd9c37f50c2b3f924d => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:42.162081 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aced4d7c-e4dc-4dbc-92df-c54659c77c4a admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:42.163702 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-aced4d7c-e4dc-4dbc-92df-c54659c77c4a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:42.176615 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aced4d7c-e4dc-4dbc-92df-c54659c77c4a admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:42.238761 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e7653ea-a07e-40dc-a9eb-76a352c1ddda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:42.239392 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e7653ea-a07e-40dc-a9eb-76a352c1ddda admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:42.239669 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e7653ea-a07e-40dc-a9eb-76a352c1ddda admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:42.239939 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e7653ea-a07e-40dc-a9eb-76a352c1ddda admin admin] PATH_INFO: `/v3/projects/0d53076c86a64a82a403402e1bb4f64b` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:42.241065 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e7653ea-a07e-40dc-a9eb-76a352c1ddda admin admin] RBAC: Authorizing `identity:delete_project(project_id=0d53076c86a64a82a403402e1bb4f64b)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:42.243552 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e7653ea-a07e-40dc-a9eb-76a352c1ddda admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:42.267760 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:42.297455 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-0e7653ea-a07e-40dc-a9eb-76a352c1ddda admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0d53076c86a64a82a403402e1bb4f64b', 'request_id': 'req-0e7653ea-a07e-40dc-a9eb-76a352c1ddda'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:42.305150 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-0e7653ea-a07e-40dc-a9eb-76a352c1ddda admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0d53076c86a64a82a403402e1bb4f64b', 'request_id': 'req-0e7653ea-a07e-40dc-a9eb-76a352c1ddda'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:42.312928 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-0e7653ea-a07e-40dc-a9eb-76a352c1ddda admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0d53076c86a64a82a403402e1bb4f64b', 'request_id': 'req-0e7653ea-a07e-40dc-a9eb-76a352c1ddda'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:42.330111 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-0e7653ea-a07e-40dc-a9eb-76a352c1ddda admin admin] The token cache is being invalidate because project 0d53076c86a64a82a403402e1bb4f64b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81562) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:01:42.330111 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-0e7653ea-a07e-40dc-a9eb-76a352c1ddda admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:42.330771 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-0e7653ea-a07e-40dc-a9eb-76a352c1ddda admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:42.332529 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 875/1752] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:01:42 2022] DELETE /identity/v3/projects/0d53076c86a64a82a403402e1bb4f64b => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:01:42.339321 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a3d6ff-9534-4da4-ac08-30845138ac29 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:42.339857 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a3d6ff-9534-4da4-ac08-30845138ac29 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:42.340342 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a3d6ff-9534-4da4-ac08-30845138ac29 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:42.372105 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-a4a3d6ff-9534-4da4-ac08-30845138ac29 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:42.373327 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-a4a3d6ff-9534-4da4-ac08-30845138ac29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:42.446890 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:42.495132 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 878/1753] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:42.495988 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f3c0cc-32cd-4053-be7f-7a85d0ec763e None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:42.495988 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f3c0cc-32cd-4053-be7f-7a85d0ec763e None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:42.495988 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f3c0cc-32cd-4053-be7f-7a85d0ec763e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:42.512739 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-d4f3c0cc-32cd-4053-be7f-7a85d0ec763e None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:42.514117 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-d4f3c0cc-32cd-4053-be7f-7a85d0ec763e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:42.526535 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 876/1754] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 20 09:01:42.529267 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a3d6ff-9534-4da4-ac08-30845138ac29 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:42.540567 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a3d6ff-9534-4da4-ac08-30845138ac29 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:42.545563 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-506bb696-6879-4f21-b8a4-e634c3582659 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:42.546370 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-506bb696-6879-4f21-b8a4-e634c3582659 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:42.546745 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-506bb696-6879-4f21-b8a4-e634c3582659 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:42.547028 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-506bb696-6879-4f21-b8a4-e634c3582659 admin admin] PATH_INFO: `/v3/domains` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:42.547689 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-506bb696-6879-4f21-b8a4-e634c3582659 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 09:01:42.547996 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-506bb696-6879-4f21-b8a4-e634c3582659 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:42.549822 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-506bb696-6879-4f21-b8a4-e634c3582659 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:42.563050 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 879/1755] 10.222.0.36 () {58 vars in 1153 bytes} [Thu Oct 20 09:01:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 09:01:42.567631 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4f3c0cc-32cd-4053-be7f-7a85d0ec763e None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:42.577419 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4f3c0cc-32cd-4053-be7f-7a85d0ec763e None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:42.582266 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aefb365-36e2-4c3c-947c-ff0344ca6370 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:42.582902 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aefb365-36e2-4c3c-947c-ff0344ca6370 admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:42.583181 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aefb365-36e2-4c3c-947c-ff0344ca6370 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:42.583448 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aefb365-36e2-4c3c-947c-ff0344ca6370 admin admin] PATH_INFO: `/v3/projects` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:42.584079 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aefb365-36e2-4c3c-947c-ff0344ca6370 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:42.586562 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aefb365-36e2-4c3c-947c-ff0344ca6370 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:42.614746 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 877/1756] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:01:42 2022] POST /identity/v3/projects => generated 385 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:01:42.618920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-506bb696-6879-4f21-b8a4-e634c3582659 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:42.629401 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-506bb696-6879-4f21-b8a4-e634c3582659 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:42.707789 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ecf5e54-dd78-4810-b80a-783b764be080 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:42.708523 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ecf5e54-dd78-4810-b80a-783b764be080 admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:42.708697 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ecf5e54-dd78-4810-b80a-783b764be080 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:42.708942 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ecf5e54-dd78-4810-b80a-783b764be080 admin admin] PATH_INFO: `/v3/users` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:42.709567 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ecf5e54-dd78-4810-b80a-783b764be080 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:42.713902 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ecf5e54-dd78-4810-b80a-783b764be080 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:42.740248 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 880/1757] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:01:42 2022] POST /identity/v3/users => generated 293 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:01:42.741718 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aefb365-36e2-4c3c-947c-ff0344ca6370 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:42.752172 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aefb365-36e2-4c3c-947c-ff0344ca6370 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:42.757397 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6315c739-4699-45c3-b628-8c8df564374a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:42.758551 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6315c739-4699-45c3-b628-8c8df564374a admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:42.759003 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6315c739-4699-45c3-b628-8c8df564374a admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:42.759346 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6315c739-4699-45c3-b628-8c8df564374a admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:42.760015 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6315c739-4699-45c3-b628-8c8df564374a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:42.761685 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6315c739-4699-45c3-b628-8c8df564374a admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:42.772316 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 878/1758] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:01:42 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 20 09:01:42.777129 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ecf5e54-dd78-4810-b80a-783b764be080 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:42.788116 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ecf5e54-dd78-4810-b80a-783b764be080 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:42.792840 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff03c66b-bfff-4b2d-a925-e2de074cfafe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:42.793438 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff03c66b-bfff-4b2d-a925-e2de074cfafe admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:42.793717 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff03c66b-bfff-4b2d-a925-e2de074cfafe admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:42.793987 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff03c66b-bfff-4b2d-a925-e2de074cfafe admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:42.794740 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff03c66b-bfff-4b2d-a925-e2de074cfafe admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:42.796419 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff03c66b-bfff-4b2d-a925-e2de074cfafe admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:42.809883 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 881/1759] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:01:42 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:01:42.811078 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6315c739-4699-45c3-b628-8c8df564374a admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:42.821395 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6315c739-4699-45c3-b628-8c8df564374a admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:42.826034 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32c1bc65-ef7e-428b-b059-b59d6051a996 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:42.827015 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c1bc65-ef7e-428b-b059-b59d6051a996 admin admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:42.830382 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c1bc65-ef7e-428b-b059-b59d6051a996 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:42.830777 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c1bc65-ef7e-428b-b059-b59d6051a996 admin admin] PATH_INFO: `/v3/projects/b4e516e9d13841a480ccd13a4ae5acd0/users/41b0b3fc0f4c4d11aca2aeb28ecb7d7e/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:42.847925 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32c1bc65-ef7e-428b-b059-b59d6051a996 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b4e516e9d13841a480ccd13a4ae5acd0, user_id=41b0b3fc0f4c4d11aca2aeb28ecb7d7e, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:42.851865 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32c1bc65-ef7e-428b-b059-b59d6051a996 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:42.869507 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 879/1760] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:01:42 2022] PUT /identity/v3/projects/b4e516e9d13841a480ccd13a4ae5acd0/users/41b0b3fc0f4c4d11aca2aeb28ecb7d7e/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:42.881781 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c42e8e6-17c0-48b0-a9d6-9fc7fa41c50a None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:42.882166 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c42e8e6-17c0-48b0-a9d6-9fc7fa41c50a None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:42.882608 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c42e8e6-17c0-48b0-a9d6-9fc7fa41c50a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:42.931940 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-8c42e8e6-17c0-48b0-a9d6-9fc7fa41c50a None None] MFA Rules not processed for user `41b0b3fc0f4c4d11aca2aeb28ecb7d7e`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:42.933347 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-8c42e8e6-17c0-48b0-a9d6-9fc7fa41c50a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:43.045954 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 882/1761] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:42 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:43.050652 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32c1bc65-ef7e-428b-b059-b59d6051a996 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:43.118035 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32c1bc65-ef7e-428b-b059-b59d6051a996 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:43.122641 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fe11b11-60e6-46b4-9d79-dd49588eba0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:43.123255 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe11b11-60e6-46b4-9d79-dd49588eba0b admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:43.123528 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe11b11-60e6-46b4-9d79-dd49588eba0b admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:43.123792 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe11b11-60e6-46b4-9d79-dd49588eba0b admin admin] PATH_INFO: `/v3/projects` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:43.124405 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fe11b11-60e6-46b4-9d79-dd49588eba0b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:43.126318 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fe11b11-60e6-46b4-9d79-dd49588eba0b admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:43.152874 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 880/1762] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:01:43 2022] POST /identity/v3/projects => generated 385 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:01:43.157046 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c42e8e6-17c0-48b0-a9d6-9fc7fa41c50a None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:43.167407 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c42e8e6-17c0-48b0-a9d6-9fc7fa41c50a None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:43.211022 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:43.224284 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f640f80-dc91-425f-84e5-d1c78cb4c93d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:43.224887 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f640f80-dc91-425f-84e5-d1c78cb4c93d admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:43.225163 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f640f80-dc91-425f-84e5-d1c78cb4c93d admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:43.225436 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f640f80-dc91-425f-84e5-d1c78cb4c93d admin admin] PATH_INFO: `/v3/users` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:43.226068 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f640f80-dc91-425f-84e5-d1c78cb4c93d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:43.228020 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f640f80-dc91-425f-84e5-d1c78cb4c93d admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:43.249152 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 883/1763] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:01:43 2022] POST /identity/v3/users => generated 299 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:01:43.253257 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fe11b11-60e6-46b4-9d79-dd49588eba0b admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:43.264176 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fe11b11-60e6-46b4-9d79-dd49588eba0b admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:43.268750 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c400e9e8-41c7-4116-9bd5-1fb611a57a10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:43.269334 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c400e9e8-41c7-4116-9bd5-1fb611a57a10 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:43.269607 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c400e9e8-41c7-4116-9bd5-1fb611a57a10 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:43.269870 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c400e9e8-41c7-4116-9bd5-1fb611a57a10 admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:43.270524 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c400e9e8-41c7-4116-9bd5-1fb611a57a10 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:43.272173 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c400e9e8-41c7-4116-9bd5-1fb611a57a10 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:43.282527 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 881/1764] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:01:43 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:01:43.286601 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f640f80-dc91-425f-84e5-d1c78cb4c93d admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:43.296338 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f640f80-dc91-425f-84e5-d1c78cb4c93d admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:43.300972 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89aff18b-22ba-432a-a8e4-39d2b319c582 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:43.301609 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89aff18b-22ba-432a-a8e4-39d2b319c582 admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:43.301884 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89aff18b-22ba-432a-a8e4-39d2b319c582 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:43.302156 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89aff18b-22ba-432a-a8e4-39d2b319c582 admin admin] PATH_INFO: `/v3/projects/56353c9fefe44e2d99ea268d5d05e314/users/bf7195dce66243f488b833e2197db2a6/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:43.317760 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89aff18b-22ba-432a-a8e4-39d2b319c582 admin admin] RBAC: Authorizing `identity:create_grant(project_id=56353c9fefe44e2d99ea268d5d05e314, user_id=bf7195dce66243f488b833e2197db2a6, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:43.321232 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89aff18b-22ba-432a-a8e4-39d2b319c582 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:43.333702 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 884/1765] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:01:43 2022] PUT /identity/v3/projects/56353c9fefe44e2d99ea268d5d05e314/users/bf7195dce66243f488b833e2197db2a6/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:43.337632 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c400e9e8-41c7-4116-9bd5-1fb611a57a10 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:43.347755 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c400e9e8-41c7-4116-9bd5-1fb611a57a10 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:43.364210 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:DELETE=7,keystone:SELECT=89,keystone:INSERT=4 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:43.406563 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bbe0fbe-85b7-4f34-a445-b8b80c8a7317 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:43.407167 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bbe0fbe-85b7-4f34-a445-b8b80c8a7317 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:43.407447 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bbe0fbe-85b7-4f34-a445-b8b80c8a7317 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:43.407718 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bbe0fbe-85b7-4f34-a445-b8b80c8a7317 admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:43.408350 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bbe0fbe-85b7-4f34-a445-b8b80c8a7317 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:43.409939 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bbe0fbe-85b7-4f34-a445-b8b80c8a7317 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:43.420301 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 882/1766] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:01:43 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:01:43.424461 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89aff18b-22ba-432a-a8e4-39d2b319c582 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:43.435329 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89aff18b-22ba-432a-a8e4-39d2b319c582 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:43.439990 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a59a0fb-81fe-40e7-98e8-96daa61de7a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:43.440638 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a59a0fb-81fe-40e7-98e8-96daa61de7a6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:43.440912 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a59a0fb-81fe-40e7-98e8-96daa61de7a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:43.441176 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a59a0fb-81fe-40e7-98e8-96daa61de7a6 admin admin] PATH_INFO: `/v3/projects/56353c9fefe44e2d99ea268d5d05e314/users/bf7195dce66243f488b833e2197db2a6/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:43.443070 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a59a0fb-81fe-40e7-98e8-96daa61de7a6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=56353c9fefe44e2d99ea268d5d05e314, user_id=bf7195dce66243f488b833e2197db2a6, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:43.445711 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a59a0fb-81fe-40e7-98e8-96daa61de7a6 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:43.456868 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 885/1767] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:01:43 2022] PUT /identity/v3/projects/56353c9fefe44e2d99ea268d5d05e314/users/bf7195dce66243f488b833e2197db2a6/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:43.467155 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3db5a98-45ff-4ab3-a195-ec6ace020aa9 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:43.467378 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3db5a98-45ff-4ab3-a195-ec6ace020aa9 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:43.467677 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3db5a98-45ff-4ab3-a195-ec6ace020aa9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:43.495727 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-a3db5a98-45ff-4ab3-a195-ec6ace020aa9 None None] MFA Rules not processed for user `bf7195dce66243f488b833e2197db2a6`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:43.496931 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-a3db5a98-45ff-4ab3-a195-ec6ace020aa9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:43.574676 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 883/1768] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:43 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:43.586843 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a59a0fb-81fe-40e7-98e8-96daa61de7a6 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:43.588586 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-7a59a0fb-81fe-40e7-98e8-96daa61de7a6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:43.661181 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a59a0fb-81fe-40e7-98e8-96daa61de7a6 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:43.665444 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a976654a-9321-46e8-998e-a68b0928dc1b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:43.666643 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a976654a-9321-46e8-998e-a68b0928dc1b service cinder] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:43.666643 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a976654a-9321-46e8-998e-a68b0928dc1b service cinder] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:43.666643 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a976654a-9321-46e8-998e-a68b0928dc1b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:43.737864 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a976654a-9321-46e8-998e-a68b0928dc1b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:43.737864 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a976654a-9321-46e8-998e-a68b0928dc1b service cinder] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:43.739053 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 886/1769] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:01:43 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 153 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 20 09:01:48.194894 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3db5a98-45ff-4ab3-a195-ec6ace020aa9 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:48.196328 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-a3db5a98-45ff-4ab3-a195-ec6ace020aa9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:48.208048 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3db5a98-45ff-4ab3-a195-ec6ace020aa9 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:48.212285 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-105d482e-76c2-4eea-baee-2ef2f318743e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:48.212888 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-105d482e-76c2-4eea-baee-2ef2f318743e service cinder] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:48.213156 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-105d482e-76c2-4eea-baee-2ef2f318743e service cinder] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:48.213516 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-105d482e-76c2-4eea-baee-2ef2f318743e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:48.223546 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-105d482e-76c2-4eea-baee-2ef2f318743e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:48.224969 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-105d482e-76c2-4eea-baee-2ef2f318743e service cinder] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:48.235221 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 884/1770] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:01:48 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:01:48.338173 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a976654a-9321-46e8-998e-a68b0928dc1b service cinder] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:48.339694 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-a976654a-9321-46e8-998e-a68b0928dc1b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:48.464863 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:48.489614 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a976654a-9321-46e8-998e-a68b0928dc1b service cinder] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:48.499194 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fac613e6-40ee-4461-bd9c-a270d3a1ec0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:48.499939 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac613e6-40ee-4461-bd9c-a270d3a1ec0c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:48.500408 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac613e6-40ee-4461-bd9c-a270d3a1ec0c admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:48.500617 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac613e6-40ee-4461-bd9c-a270d3a1ec0c admin admin] PATH_INFO: `/v3/users/e95df59aa56349558136c7e5f4858427` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:48.501838 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fac613e6-40ee-4461-bd9c-a270d3a1ec0c admin admin] RBAC: Authorizing `identity:delete_user(user_id=e95df59aa56349558136c7e5f4858427)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:48.503964 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fac613e6-40ee-4461-bd9c-a270d3a1ec0c admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:48.564907 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-fac613e6-40ee-4461-bd9c-a270d3a1ec0c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e95df59aa56349558136c7e5f4858427'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:48.586998 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-fac613e6-40ee-4461-bd9c-a270d3a1ec0c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e95df59aa56349558136c7e5f4858427'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:48.600658 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-fac613e6-40ee-4461-bd9c-a270d3a1ec0c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e95df59aa56349558136c7e5f4858427'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:48.611042 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-fac613e6-40ee-4461-bd9c-a270d3a1ec0c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e95df59aa56349558136c7e5f4858427'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:48.613683 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-fac613e6-40ee-4461-bd9c-a270d3a1ec0c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e95df59aa56349558136c7e5f4858427'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:48.625813 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 887/1771] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:01:48 2022] DELETE /identity/v3/users/e95df59aa56349558136c7e5f4858427 => generated 0 bytes in 285 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:48.626839 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3427cc96-cab6-4e17-9b73-09efd81fc3c4 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:48.627291 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3427cc96-cab6-4e17-9b73-09efd81fc3c4 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:48.627570 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3427cc96-cab6-4e17-9b73-09efd81fc3c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:48.647992 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-3427cc96-cab6-4e17-9b73-09efd81fc3c4 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:48.649733 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-3427cc96-cab6-4e17-9b73-09efd81fc3c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:48.712998 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 885/1772] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 20 09:01:48.723153 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fac613e6-40ee-4461-bd9c-a270d3a1ec0c admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:48.724578 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-fac613e6-40ee-4461-bd9c-a270d3a1ec0c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:48.783347 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fac613e6-40ee-4461-bd9c-a270d3a1ec0c admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:48.786986 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2670f519-f327-4044-b8a0-ed883c3e72f9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:48.787596 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2670f519-f327-4044-b8a0-ed883c3e72f9 service neutron] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:48.787869 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2670f519-f327-4044-b8a0-ed883c3e72f9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:48.788143 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2670f519-f327-4044-b8a0-ed883c3e72f9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:48.797237 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2670f519-f327-4044-b8a0-ed883c3e72f9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:48.798886 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2670f519-f327-4044-b8a0-ed883c3e72f9 service neutron] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:48.808317 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 888/1773] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:01:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:01:49.072061 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3427cc96-cab6-4e17-9b73-09efd81fc3c4 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:49.073593 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-3427cc96-cab6-4e17-9b73-09efd81fc3c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:49.096874 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3427cc96-cab6-4e17-9b73-09efd81fc3c4 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:49.103137 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8516817a-f2c5-4f2d-afc8-ae4bf4f6f80b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:49.103665 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8516817a-f2c5-4f2d-afc8-ae4bf4f6f80b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:49.103938 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8516817a-f2c5-4f2d-afc8-ae4bf4f6f80b admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:49.104203 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8516817a-f2c5-4f2d-afc8-ae4bf4f6f80b admin admin] PATH_INFO: `/v3/projects/a91bb0a442ec4534ba30082b7be8dbf5` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:49.105433 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8516817a-f2c5-4f2d-afc8-ae4bf4f6f80b admin admin] RBAC: Authorizing `identity:delete_project(project_id=a91bb0a442ec4534ba30082b7be8dbf5)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:49.107583 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8516817a-f2c5-4f2d-afc8-ae4bf4f6f80b admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:49.136889 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:49.160716 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-8516817a-f2c5-4f2d-afc8-ae4bf4f6f80b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a91bb0a442ec4534ba30082b7be8dbf5', 'request_id': 'req-8516817a-f2c5-4f2d-afc8-ae4bf4f6f80b'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:49.167577 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-8516817a-f2c5-4f2d-afc8-ae4bf4f6f80b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a91bb0a442ec4534ba30082b7be8dbf5', 'request_id': 'req-8516817a-f2c5-4f2d-afc8-ae4bf4f6f80b'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:49.173032 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-8516817a-f2c5-4f2d-afc8-ae4bf4f6f80b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a91bb0a442ec4534ba30082b7be8dbf5', 'request_id': 'req-8516817a-f2c5-4f2d-afc8-ae4bf4f6f80b'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:49.185052 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-8516817a-f2c5-4f2d-afc8-ae4bf4f6f80b admin admin] The token cache is being invalidate because project a91bb0a442ec4534ba30082b7be8dbf5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81562) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:01:49.185330 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-8516817a-f2c5-4f2d-afc8-ae4bf4f6f80b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:49.185939 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-8516817a-f2c5-4f2d-afc8-ae4bf4f6f80b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:49.187430 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 886/1774] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:01:49 2022] DELETE /identity/v3/projects/a91bb0a442ec4534ba30082b7be8dbf5 => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:49.192350 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91cdf47a-8947-45ba-acdc-95c91d9917e4 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:49.192662 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91cdf47a-8947-45ba-acdc-95c91d9917e4 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:49.192957 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91cdf47a-8947-45ba-acdc-95c91d9917e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:49.212896 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-91cdf47a-8947-45ba-acdc-95c91d9917e4 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:49.214077 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-91cdf47a-8947-45ba-acdc-95c91d9917e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:49.272525 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 889/1775] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:49.280893 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5589c8c-0bbd-4da5-b5cb-31288707dead None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:49.281350 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5589c8c-0bbd-4da5-b5cb-31288707dead None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:49.281436 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5589c8c-0bbd-4da5-b5cb-31288707dead None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:49.301768 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-f5589c8c-0bbd-4da5-b5cb-31288707dead None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:49.302991 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-f5589c8c-0bbd-4da5-b5cb-31288707dead None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:49.310690 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 887/1776] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:49.316347 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91cdf47a-8947-45ba-acdc-95c91d9917e4 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:49.326022 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91cdf47a-8947-45ba-acdc-95c91d9917e4 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:49.330725 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92e119e1-076d-4ba5-bc38-bb235265515a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:49.331323 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e119e1-076d-4ba5-bc38-bb235265515a admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:49.331602 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e119e1-076d-4ba5-bc38-bb235265515a admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:49.331873 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e119e1-076d-4ba5-bc38-bb235265515a admin admin] PATH_INFO: `/v3/domains` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:49.332519 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92e119e1-076d-4ba5-bc38-bb235265515a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 09:01:49.332821 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92e119e1-076d-4ba5-bc38-bb235265515a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:49.334494 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92e119e1-076d-4ba5-bc38-bb235265515a admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:49.346893 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 890/1777] 10.222.0.36 () {58 vars in 1153 bytes} [Thu Oct 20 09:01:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 09:01:49.351977 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5589c8c-0bbd-4da5-b5cb-31288707dead None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:49.361315 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5589c8c-0bbd-4da5-b5cb-31288707dead None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:49.365942 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8282818d-fd3b-42c2-afa2-10def6068654 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:49.366574 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8282818d-fd3b-42c2-afa2-10def6068654 admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:49.366965 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8282818d-fd3b-42c2-afa2-10def6068654 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:49.367262 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8282818d-fd3b-42c2-afa2-10def6068654 admin admin] PATH_INFO: `/v3/projects` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:49.367904 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8282818d-fd3b-42c2-afa2-10def6068654 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:49.369805 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8282818d-fd3b-42c2-afa2-10def6068654 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:49.396792 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 888/1778] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:01:49 2022] POST /identity/v3/projects => generated 373 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:01:49.401707 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92e119e1-076d-4ba5-bc38-bb235265515a admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:49.406888 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:49.412860 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92e119e1-076d-4ba5-bc38-bb235265515a admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:49.471409 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2791c02b-ed09-4808-aa6f-52dcaeabd233 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:49.471705 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2791c02b-ed09-4808-aa6f-52dcaeabd233 admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:49.471986 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2791c02b-ed09-4808-aa6f-52dcaeabd233 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:49.472255 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2791c02b-ed09-4808-aa6f-52dcaeabd233 admin admin] PATH_INFO: `/v3/users` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:49.472876 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2791c02b-ed09-4808-aa6f-52dcaeabd233 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:49.474842 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2791c02b-ed09-4808-aa6f-52dcaeabd233 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:49.499875 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 891/1779] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:01:49 2022] POST /identity/v3/users => generated 287 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:01:49.503941 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8282818d-fd3b-42c2-afa2-10def6068654 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:49.516288 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8282818d-fd3b-42c2-afa2-10def6068654 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:49.521271 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15ab2b3d-ee85-404c-bd86-3c1f2785ae89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:49.521970 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ab2b3d-ee85-404c-bd86-3c1f2785ae89 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:49.522218 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ab2b3d-ee85-404c-bd86-3c1f2785ae89 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:49.522512 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ab2b3d-ee85-404c-bd86-3c1f2785ae89 admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:49.523911 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15ab2b3d-ee85-404c-bd86-3c1f2785ae89 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:49.526416 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15ab2b3d-ee85-404c-bd86-3c1f2785ae89 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:49.537698 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 889/1780] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:01:49 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:01:49.547776 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2791c02b-ed09-4808-aa6f-52dcaeabd233 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:49.559997 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2791c02b-ed09-4808-aa6f-52dcaeabd233 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:49.566419 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96e8a08c-cf2f-404c-afa1-630f37b487a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:49.567201 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e8a08c-cf2f-404c-afa1-630f37b487a0 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:49.567574 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e8a08c-cf2f-404c-afa1-630f37b487a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:49.567920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e8a08c-cf2f-404c-afa1-630f37b487a0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:49.568640 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96e8a08c-cf2f-404c-afa1-630f37b487a0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:49.571062 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96e8a08c-cf2f-404c-afa1-630f37b487a0 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:49.590945 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 892/1781] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:01:49 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:01:49.591232 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15ab2b3d-ee85-404c-bd86-3c1f2785ae89 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:49.604867 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15ab2b3d-ee85-404c-bd86-3c1f2785ae89 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:49.611794 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6fc51dd-a5ba-4e52-8250-6d6692e1584d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:49.612741 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6fc51dd-a5ba-4e52-8250-6d6692e1584d admin admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:49.613022 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6fc51dd-a5ba-4e52-8250-6d6692e1584d admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:49.613459 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6fc51dd-a5ba-4e52-8250-6d6692e1584d admin admin] PATH_INFO: `/v3/projects/cb86b3c2194e49d4bd586a52c5660965/users/b611cd0d7d2b43959421fc8bdc39bd13/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:49.641824 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6fc51dd-a5ba-4e52-8250-6d6692e1584d admin admin] RBAC: Authorizing `identity:create_grant(project_id=cb86b3c2194e49d4bd586a52c5660965, user_id=b611cd0d7d2b43959421fc8bdc39bd13, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:49.645639 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6fc51dd-a5ba-4e52-8250-6d6692e1584d admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:49.670278 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 890/1782] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:01:49 2022] PUT /identity/v3/projects/cb86b3c2194e49d4bd586a52c5660965/users/b611cd0d7d2b43959421fc8bdc39bd13/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 80 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:01:49.682650 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40230ba-39dc-4762-9be8-74ca6b37d580 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:49.683181 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40230ba-39dc-4762-9be8-74ca6b37d580 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:49.683638 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40230ba-39dc-4762-9be8-74ca6b37d580 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:49.717978 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-a40230ba-39dc-4762-9be8-74ca6b37d580 None None] MFA Rules not processed for user `b611cd0d7d2b43959421fc8bdc39bd13`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:49.719224 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-a40230ba-39dc-4762-9be8-74ca6b37d580 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:49.801491 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 893/1783] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:49 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:49.854512 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6fc51dd-a5ba-4e52-8250-6d6692e1584d admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:49.862263 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-c6fc51dd-a5ba-4e52-8250-6d6692e1584d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:49.932166 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6fc51dd-a5ba-4e52-8250-6d6692e1584d admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:49.933793 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95dd2f90-617c-4794-91b8-25ddb2eee3c4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:49.934423 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95dd2f90-617c-4794-91b8-25ddb2eee3c4 service cinder] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:49.934733 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95dd2f90-617c-4794-91b8-25ddb2eee3c4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:49.935005 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95dd2f90-617c-4794-91b8-25ddb2eee3c4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:49.945023 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95dd2f90-617c-4794-91b8-25ddb2eee3c4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:49.946489 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95dd2f90-617c-4794-91b8-25ddb2eee3c4 service cinder] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:49.956604 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 891/1784] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:01:49 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:01:49.984075 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a40230ba-39dc-4762-9be8-74ca6b37d580 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:50.109406 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a40230ba-39dc-4762-9be8-74ca6b37d580 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:50.118496 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a4d968-fe18-444c-b2a9-5cd81fadb3c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:50.119320 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a4d968-fe18-444c-b2a9-5cd81fadb3c5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:50.119604 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a4d968-fe18-444c-b2a9-5cd81fadb3c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:50.119872 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a4d968-fe18-444c-b2a9-5cd81fadb3c5 admin admin] PATH_INFO: `/v3/users/b611cd0d7d2b43959421fc8bdc39bd13` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:50.123682 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49a4d968-fe18-444c-b2a9-5cd81fadb3c5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b611cd0d7d2b43959421fc8bdc39bd13)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:50.128449 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49a4d968-fe18-444c-b2a9-5cd81fadb3c5 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:50.153196 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=96,keystone:INSERT=3,keystone:DELETE=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:50.203997 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-49a4d968-fe18-444c-b2a9-5cd81fadb3c5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b611cd0d7d2b43959421fc8bdc39bd13'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:50.216421 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-49a4d968-fe18-444c-b2a9-5cd81fadb3c5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b611cd0d7d2b43959421fc8bdc39bd13'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:50.236895 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-49a4d968-fe18-444c-b2a9-5cd81fadb3c5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b611cd0d7d2b43959421fc8bdc39bd13'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:50.251481 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-49a4d968-fe18-444c-b2a9-5cd81fadb3c5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b611cd0d7d2b43959421fc8bdc39bd13'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:50.253308 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-49a4d968-fe18-444c-b2a9-5cd81fadb3c5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b611cd0d7d2b43959421fc8bdc39bd13'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:50.284023 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 894/1785] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:01:49 2022] DELETE /identity/v3/users/b611cd0d7d2b43959421fc8bdc39bd13 => generated 0 bytes in 301 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:50.290077 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e67569-063c-4135-ab0a-ff0c95b388a5 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:50.290987 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e67569-063c-4135-ab0a-ff0c95b388a5 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:50.291438 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e67569-063c-4135-ab0a-ff0c95b388a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:50.313373 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-49e67569-063c-4135-ab0a-ff0c95b388a5 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:50.315323 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-49e67569-063c-4135-ab0a-ff0c95b388a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:50.382446 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=91,keystone:DELETE=6,keystone:INSERT=3 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:50.410179 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 892/1786] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 20 09:01:50.425076 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a4d968-fe18-444c-b2a9-5cd81fadb3c5 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:50.427825 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-49a4d968-fe18-444c-b2a9-5cd81fadb3c5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:50.513908 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a4d968-fe18-444c-b2a9-5cd81fadb3c5 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:50.520023 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7729cc56-10c4-454a-a8af-f1dde6f9e929 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:50.521871 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7729cc56-10c4-454a-a8af-f1dde6f9e929 service neutron] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:50.522159 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7729cc56-10c4-454a-a8af-f1dde6f9e929 service neutron] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:50.522705 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7729cc56-10c4-454a-a8af-f1dde6f9e929 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:50.535979 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7729cc56-10c4-454a-a8af-f1dde6f9e929 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:50.537630 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7729cc56-10c4-454a-a8af-f1dde6f9e929 service neutron] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:50.551950 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 895/1787] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:01:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:01:50.834744 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49e67569-063c-4135-ab0a-ff0c95b388a5 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:50.836251 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-49e67569-063c-4135-ab0a-ff0c95b388a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:50.854006 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49e67569-063c-4135-ab0a-ff0c95b388a5 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:50.858746 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-799333a2-89ed-47dc-ad56-d774b02ee5c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:50.859597 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799333a2-89ed-47dc-ad56-d774b02ee5c6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:50.859769 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799333a2-89ed-47dc-ad56-d774b02ee5c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:50.860063 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799333a2-89ed-47dc-ad56-d774b02ee5c6 admin admin] PATH_INFO: `/v3/projects/cb86b3c2194e49d4bd586a52c5660965` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:50.861362 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-799333a2-89ed-47dc-ad56-d774b02ee5c6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=cb86b3c2194e49d4bd586a52c5660965)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:50.863397 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-799333a2-89ed-47dc-ad56-d774b02ee5c6 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:50.916317 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-799333a2-89ed-47dc-ad56-d774b02ee5c6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cb86b3c2194e49d4bd586a52c5660965', 'request_id': 'req-799333a2-89ed-47dc-ad56-d774b02ee5c6'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:50.923125 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-799333a2-89ed-47dc-ad56-d774b02ee5c6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cb86b3c2194e49d4bd586a52c5660965', 'request_id': 'req-799333a2-89ed-47dc-ad56-d774b02ee5c6'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:50.930091 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-799333a2-89ed-47dc-ad56-d774b02ee5c6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cb86b3c2194e49d4bd586a52c5660965', 'request_id': 'req-799333a2-89ed-47dc-ad56-d774b02ee5c6'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:50.946789 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-799333a2-89ed-47dc-ad56-d774b02ee5c6 admin admin] The token cache is being invalidate because project cb86b3c2194e49d4bd586a52c5660965 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81562) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:01:50.946789 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-799333a2-89ed-47dc-ad56-d774b02ee5c6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:50.946789 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-799333a2-89ed-47dc-ad56-d774b02ee5c6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:50.948565 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 893/1788] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:01:50 2022] DELETE /identity/v3/projects/cb86b3c2194e49d4bd586a52c5660965 => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:01:50.953726 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66bf325b-61c6-4d58-b8b2-0b8b31f419c7 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:50.954012 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66bf325b-61c6-4d58-b8b2-0b8b31f419c7 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:50.954278 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66bf325b-61c6-4d58-b8b2-0b8b31f419c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:50.975592 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-66bf325b-61c6-4d58-b8b2-0b8b31f419c7 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:50.977352 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-66bf325b-61c6-4d58-b8b2-0b8b31f419c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:51.055193 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 896/1789] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:51.063228 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5f7587-cb62-4c2a-80eb-2c43148f777f None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:51.063522 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5f7587-cb62-4c2a-80eb-2c43148f777f None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:51.063797 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5f7587-cb62-4c2a-80eb-2c43148f777f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:51.085135 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-3f5f7587-cb62-4c2a-80eb-2c43148f777f None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:51.086982 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-3f5f7587-cb62-4c2a-80eb-2c43148f777f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:51.095128 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 894/1790] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 20 09:01:51.099485 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66bf325b-61c6-4d58-b8b2-0b8b31f419c7 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:51.109205 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66bf325b-61c6-4d58-b8b2-0b8b31f419c7 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:51.113578 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3ee9315-7477-40ba-abf0-49cca1f8f480 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:51.114173 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3ee9315-7477-40ba-abf0-49cca1f8f480 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:51.114445 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3ee9315-7477-40ba-abf0-49cca1f8f480 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:51.114758 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3ee9315-7477-40ba-abf0-49cca1f8f480 admin admin] PATH_INFO: `/v3/domains` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:51.115492 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3ee9315-7477-40ba-abf0-49cca1f8f480 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 09:01:51.115688 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3ee9315-7477-40ba-abf0-49cca1f8f480 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:51.117298 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3ee9315-7477-40ba-abf0-49cca1f8f480 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:51.133682 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 897/1791] 10.222.0.36 () {58 vars in 1153 bytes} [Thu Oct 20 09:01:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 09:01:51.135423 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f5f7587-cb62-4c2a-80eb-2c43148f777f None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:51.145699 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f5f7587-cb62-4c2a-80eb-2c43148f777f None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:51.151951 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd4be7e-3ef8-423d-ae53-966c6d66550f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:51.152586 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd4be7e-3ef8-423d-ae53-966c6d66550f admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:51.152905 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd4be7e-3ef8-423d-ae53-966c6d66550f admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:51.153257 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd4be7e-3ef8-423d-ae53-966c6d66550f admin admin] PATH_INFO: `/v3/projects` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:51.153828 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfd4be7e-3ef8-423d-ae53-966c6d66550f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:51.162898 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfd4be7e-3ef8-423d-ae53-966c6d66550f admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:51.192178 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 895/1792] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:01:51 2022] POST /identity/v3/projects => generated 383 bytes in 58 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:01:51.197197 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3ee9315-7477-40ba-abf0-49cca1f8f480 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:51.207972 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3ee9315-7477-40ba-abf0-49cca1f8f480 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:51.230401 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:51.287839 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26b19270-c58b-4299-83f5-7e4453d28c2c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:51.289064 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b19270-c58b-4299-83f5-7e4453d28c2c admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:51.289064 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b19270-c58b-4299-83f5-7e4453d28c2c admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:51.289337 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b19270-c58b-4299-83f5-7e4453d28c2c admin admin] PATH_INFO: `/v3/users` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:51.290311 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26b19270-c58b-4299-83f5-7e4453d28c2c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:51.292569 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26b19270-c58b-4299-83f5-7e4453d28c2c admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:51.319049 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 898/1793] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:01:51 2022] POST /identity/v3/users => generated 292 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:01:51.323714 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd4be7e-3ef8-423d-ae53-966c6d66550f admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:51.333959 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd4be7e-3ef8-423d-ae53-966c6d66550f admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:51.338557 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-857b7561-9258-43d4-bee8-f6f86f8a79c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:51.339151 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857b7561-9258-43d4-bee8-f6f86f8a79c9 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:51.339424 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857b7561-9258-43d4-bee8-f6f86f8a79c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:51.339690 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857b7561-9258-43d4-bee8-f6f86f8a79c9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:51.340584 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-857b7561-9258-43d4-bee8-f6f86f8a79c9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:51.342184 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-857b7561-9258-43d4-bee8-f6f86f8a79c9 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:51.352329 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 896/1794] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:01:51 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:01:51.356093 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26b19270-c58b-4299-83f5-7e4453d28c2c admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:51.369573 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26b19270-c58b-4299-83f5-7e4453d28c2c admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:51.375506 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a54a61-99c4-4a7c-acd3-eaf9aa5458bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:51.376029 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a54a61-99c4-4a7c-acd3-eaf9aa5458bc admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:51.376377 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a54a61-99c4-4a7c-acd3-eaf9aa5458bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:51.376747 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a54a61-99c4-4a7c-acd3-eaf9aa5458bc admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:51.377456 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86a54a61-99c4-4a7c-acd3-eaf9aa5458bc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:51.379381 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86a54a61-99c4-4a7c-acd3-eaf9aa5458bc admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:51.391856 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 899/1795] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:01:51 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:01:51.396027 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-857b7561-9258-43d4-bee8-f6f86f8a79c9 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:51.407192 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-857b7561-9258-43d4-bee8-f6f86f8a79c9 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:51.411798 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3cbcea1-8e96-469e-b4c2-326395197449 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:51.412437 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3cbcea1-8e96-469e-b4c2-326395197449 admin admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:51.412898 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3cbcea1-8e96-469e-b4c2-326395197449 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:51.413173 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3cbcea1-8e96-469e-b4c2-326395197449 admin admin] PATH_INFO: `/v3/projects/9dc1a3b80bd34c8e8f691a9b47457f35/users/7e0951446ad449d6b725fe6cc14df1f2/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:51.429959 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3cbcea1-8e96-469e-b4c2-326395197449 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9dc1a3b80bd34c8e8f691a9b47457f35, user_id=7e0951446ad449d6b725fe6cc14df1f2, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:51.432787 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3cbcea1-8e96-469e-b4c2-326395197449 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:51.450258 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 897/1796] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:01:51 2022] PUT /identity/v3/projects/9dc1a3b80bd34c8e8f691a9b47457f35/users/7e0951446ad449d6b725fe6cc14df1f2/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:51.455166 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f57742-030e-4a95-b112-75a4387e767f None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:51.455635 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f57742-030e-4a95-b112-75a4387e767f None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:51.455736 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f57742-030e-4a95-b112-75a4387e767f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:51.489093 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-67f57742-030e-4a95-b112-75a4387e767f None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:51.490934 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-67f57742-030e-4a95-b112-75a4387e767f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:51.564790 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 900/1797] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:51.574728 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3cbcea1-8e96-469e-b4c2-326395197449 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:51.576765 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-c3cbcea1-8e96-469e-b4c2-326395197449 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:51.648228 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3cbcea1-8e96-469e-b4c2-326395197449 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:51.651973 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af4742ad-59f3-4f4c-ba7c-5fd53887a18c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:51.652603 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af4742ad-59f3-4f4c-ba7c-5fd53887a18c service neutron] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:51.652907 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af4742ad-59f3-4f4c-ba7c-5fd53887a18c service neutron] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:51.653172 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af4742ad-59f3-4f4c-ba7c-5fd53887a18c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:51.662275 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af4742ad-59f3-4f4c-ba7c-5fd53887a18c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:51.664283 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af4742ad-59f3-4f4c-ba7c-5fd53887a18c service neutron] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:51.673689 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 898/1798] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:01:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:01:52.283278 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d1e4aa9-d69b-43f6-a986-f995cb4a89cb None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:52.283620 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d1e4aa9-d69b-43f6-a986-f995cb4a89cb None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:52.283962 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d1e4aa9-d69b-43f6-a986-f995cb4a89cb None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:52.285567 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 901/1799] 10.222.0.36 () {54 vars in 921 bytes} [Thu Oct 20 09:01:52 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 09:01:52.559486 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb948ec-a3de-4d22-bb69-51d371f6a187 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:52.559967 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb948ec-a3de-4d22-bb69-51d371f6a187 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:52.560231 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb948ec-a3de-4d22-bb69-51d371f6a187 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:52.623101 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f615f7e-93bd-4668-8afb-5b0fd288829e None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:52.623581 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f615f7e-93bd-4668-8afb-5b0fd288829e None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:52.623661 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f615f7e-93bd-4668-8afb-5b0fd288829e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:52.642066 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:52.649414 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-fbb948ec-a3de-4d22-bb69-51d371f6a187 None None] MFA Rules not processed for user `7e0951446ad449d6b725fe6cc14df1f2`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:52.652773 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-fbb948ec-a3de-4d22-bb69-51d371f6a187 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:52.653581 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-9f615f7e-93bd-4668-8afb-5b0fd288829e None None] MFA Rules not processed for user `ad03ab76b9c44b1cb369475afa25eddd`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:52.655701 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-9f615f7e-93bd-4668-8afb-5b0fd288829e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:52.765875 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 902/1800] 10.222.0.36 () {58 vars in 1018 bytes} [Thu Oct 20 09:01:52 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:52.780636 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f615f7e-93bd-4668-8afb-5b0fd288829e None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:52.782578 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-9f615f7e-93bd-4668-8afb-5b0fd288829e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:52.790532 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 899/1801] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:52 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 232 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:52.804742 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb948ec-a3de-4d22-bb69-51d371f6a187 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:52.806166 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-fbb948ec-a3de-4d22-bb69-51d371f6a187 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:52.810638 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f615f7e-93bd-4668-8afb-5b0fd288829e None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:52.827983 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-148616f2-bfce-4d73-9655-2ee00bfb48e8 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ad03ab76b9c44b1cb369475afa25eddd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:52.828845 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148616f2-bfce-4d73-9655-2ee00bfb48e8 service placement] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:52.829217 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148616f2-bfce-4d73-9655-2ee00bfb48e8 service placement] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:52.829583 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148616f2-bfce-4d73-9655-2ee00bfb48e8 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:52.852002 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-148616f2-bfce-4d73-9655-2ee00bfb48e8 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:52.856342 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-148616f2-bfce-4d73-9655-2ee00bfb48e8 service placement] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:52.870662 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 903/1802] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:01:52 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:01:52.939651 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb948ec-a3de-4d22-bb69-51d371f6a187 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:52.943605 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b322a08-7ef4-46bc-849e-5175ffeb4ee4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:52.944528 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b322a08-7ef4-46bc-849e-5175ffeb4ee4 service cinder] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:52.945525 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b322a08-7ef4-46bc-849e-5175ffeb4ee4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:52.945938 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b322a08-7ef4-46bc-849e-5175ffeb4ee4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:52.956283 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b322a08-7ef4-46bc-849e-5175ffeb4ee4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:52.958343 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b322a08-7ef4-46bc-849e-5175ffeb4ee4 service cinder] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:52.974653 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 900/1803] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:01:52 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 171 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 20 09:01:53.411812 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7dadb3e-a3ea-4d99-bc67-1332635f9175 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:53.412076 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7dadb3e-a3ea-4d99-bc67-1332635f9175 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:53.412296 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7dadb3e-a3ea-4d99-bc67-1332635f9175 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:53.414336 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 904/1804] 10.222.0.36 () {54 vars in 921 bytes} [Thu Oct 20 09:01:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 09:01:53.426657 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c57f81-f06e-452e-967f-049ec99400f0 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:53.427020 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c57f81-f06e-452e-967f-049ec99400f0 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:53.427357 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c57f81-f06e-452e-967f-049ec99400f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:53.450853 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-c7c57f81-f06e-452e-967f-049ec99400f0 None None] MFA Rules not processed for user `ad03ab76b9c44b1cb369475afa25eddd`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:53.452084 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-c7c57f81-f06e-452e-967f-049ec99400f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:53.462083 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 901/1805] 10.222.0.36 () {58 vars in 1018 bytes} [Thu Oct 20 09:01:53 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:53.480554 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7dadb3e-a3ea-4d99-bc67-1332635f9175 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:53.498359 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7dadb3e-a3ea-4d99-bc67-1332635f9175 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:53.505963 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7c57f81-f06e-452e-967f-049ec99400f0 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:53.508169 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-c7c57f81-f06e-452e-967f-049ec99400f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:53.508690 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-594395ce-a6fa-4d30-870f-f3c6cdd8de57 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ad03ab76b9c44b1cb369475afa25eddd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:53.509660 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-594395ce-a6fa-4d30-870f-f3c6cdd8de57 service placement] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:53.510152 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-594395ce-a6fa-4d30-870f-f3c6cdd8de57 service placement] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:53.511048 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-594395ce-a6fa-4d30-870f-f3c6cdd8de57 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:53.520828 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-594395ce-a6fa-4d30-870f-f3c6cdd8de57 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:53.522608 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-594395ce-a6fa-4d30-870f-f3c6cdd8de57 service placement] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:53.522982 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7c57f81-f06e-452e-967f-049ec99400f0 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:53.528822 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6423adc-5dd4-45e4-9ef0-616e54939dfb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:53.530432 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6423adc-5dd4-45e4-9ef0-616e54939dfb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:53.530432 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6423adc-5dd4-45e4-9ef0-616e54939dfb admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:53.530432 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6423adc-5dd4-45e4-9ef0-616e54939dfb admin admin] PATH_INFO: `/v3/users/9c7a132459ba43ed87d9c429e2f59940` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:53.531783 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6423adc-5dd4-45e4-9ef0-616e54939dfb admin admin] RBAC: Authorizing `identity:delete_user(user_id=9c7a132459ba43ed87d9c429e2f59940)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:53.533957 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6423adc-5dd4-45e4-9ef0-616e54939dfb admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:53.536478 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 905/1806] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:01:53 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 56 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:01:53.586969 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-c6423adc-5dd4-45e4-9ef0-616e54939dfb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9c7a132459ba43ed87d9c429e2f59940'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:53.594025 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-c6423adc-5dd4-45e4-9ef0-616e54939dfb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9c7a132459ba43ed87d9c429e2f59940'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:53.607308 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-c6423adc-5dd4-45e4-9ef0-616e54939dfb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9c7a132459ba43ed87d9c429e2f59940'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:53.614142 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-c6423adc-5dd4-45e4-9ef0-616e54939dfb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9c7a132459ba43ed87d9c429e2f59940'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:53.614930 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-c6423adc-5dd4-45e4-9ef0-616e54939dfb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9c7a132459ba43ed87d9c429e2f59940'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:53.621444 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-594395ce-a6fa-4d30-870f-f3c6cdd8de57 service placement] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:53.622843 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 902/1807] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:01:53 2022] DELETE /identity/v3/users/9c7a132459ba43ed87d9c429e2f59940 => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:01:53.625737 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-594395ce-a6fa-4d30-870f-f3c6cdd8de57 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:53.637229 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-594395ce-a6fa-4d30-870f-f3c6cdd8de57 service placement] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:53.641998 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:53.697510 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c92d9f7-c9c0-4bff-b512-34a0a5534764 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:53.698143 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c92d9f7-c9c0-4bff-b512-34a0a5534764 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:53.698441 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c92d9f7-c9c0-4bff-b512-34a0a5534764 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:53.698737 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c92d9f7-c9c0-4bff-b512-34a0a5534764 admin admin] PATH_INFO: `/v3/users/41b0b3fc0f4c4d11aca2aeb28ecb7d7e` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:53.699878 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c92d9f7-c9c0-4bff-b512-34a0a5534764 admin admin] RBAC: Authorizing `identity:delete_user(user_id=41b0b3fc0f4c4d11aca2aeb28ecb7d7e)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:53.701867 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c92d9f7-c9c0-4bff-b512-34a0a5534764 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:53.751559 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-6c92d9f7-c9c0-4bff-b512-34a0a5534764 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '41b0b3fc0f4c4d11aca2aeb28ecb7d7e'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:53.768653 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-6c92d9f7-c9c0-4bff-b512-34a0a5534764 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '41b0b3fc0f4c4d11aca2aeb28ecb7d7e'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:53.782168 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-6c92d9f7-c9c0-4bff-b512-34a0a5534764 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '41b0b3fc0f4c4d11aca2aeb28ecb7d7e'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:53.790401 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-6c92d9f7-c9c0-4bff-b512-34a0a5534764 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '41b0b3fc0f4c4d11aca2aeb28ecb7d7e'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:53.791165 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-6c92d9f7-c9c0-4bff-b512-34a0a5534764 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '41b0b3fc0f4c4d11aca2aeb28ecb7d7e'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:53.797547 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 906/1808] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:01:53 2022] DELETE /identity/v3/users/41b0b3fc0f4c4d11aca2aeb28ecb7d7e => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:01:53.801883 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a1eb06-ccc4-47c1-93e1-bd6cfe7c3a8f None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:53.802178 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a1eb06-ccc4-47c1-93e1-bd6cfe7c3a8f None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:53.802456 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a1eb06-ccc4-47c1-93e1-bd6cfe7c3a8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:53.805938 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c92d9f7-c9c0-4bff-b512-34a0a5534764 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:53.808432 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-6c92d9f7-c9c0-4bff-b512-34a0a5534764 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:53.829300 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-e7a1eb06-ccc4-47c1-93e1-bd6cfe7c3a8f None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:53.829505 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-e7a1eb06-ccc4-47c1-93e1-bd6cfe7c3a8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:53.835944 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:53.888224 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c92d9f7-c9c0-4bff-b512-34a0a5534764 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:53.893461 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a60b7aaa-9e19-44dd-ab6d-74898ddfc242 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:53.894246 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60b7aaa-9e19-44dd-ab6d-74898ddfc242 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:53.894694 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60b7aaa-9e19-44dd-ab6d-74898ddfc242 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:53.894953 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60b7aaa-9e19-44dd-ab6d-74898ddfc242 admin admin] PATH_INFO: `/v3/projects/1f24684045924d9bbd661bd8a74f751c` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:53.896248 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a60b7aaa-9e19-44dd-ab6d-74898ddfc242 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1f24684045924d9bbd661bd8a74f751c)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:53.898946 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a60b7aaa-9e19-44dd-ab6d-74898ddfc242 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:53.903441 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 903/1809] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:53.914272 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7a1eb06-ccc4-47c1-93e1-bd6cfe7c3a8f None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:53.916608 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-e7a1eb06-ccc4-47c1-93e1-bd6cfe7c3a8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:53.950297 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:53.971366 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-a60b7aaa-9e19-44dd-ab6d-74898ddfc242 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1f24684045924d9bbd661bd8a74f751c', 'request_id': 'req-a60b7aaa-9e19-44dd-ab6d-74898ddfc242'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:53.982216 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-a60b7aaa-9e19-44dd-ab6d-74898ddfc242 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1f24684045924d9bbd661bd8a74f751c', 'request_id': 'req-a60b7aaa-9e19-44dd-ab6d-74898ddfc242'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:53.989252 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-a60b7aaa-9e19-44dd-ab6d-74898ddfc242 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1f24684045924d9bbd661bd8a74f751c', 'request_id': 'req-a60b7aaa-9e19-44dd-ab6d-74898ddfc242'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:54.010417 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-a60b7aaa-9e19-44dd-ab6d-74898ddfc242 admin admin] The token cache is being invalidate because project 1f24684045924d9bbd661bd8a74f751c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81561) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:01:54.010836 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-a60b7aaa-9e19-44dd-ab6d-74898ddfc242 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:54.012066 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-a60b7aaa-9e19-44dd-ab6d-74898ddfc242 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:54.015685 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 907/1810] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:01:53 2022] DELETE /identity/v3/projects/1f24684045924d9bbd661bd8a74f751c => generated 0 bytes in 210 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:54.019590 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a60b7aaa-9e19-44dd-ab6d-74898ddfc242 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:54.021606 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-a60b7aaa-9e19-44dd-ab6d-74898ddfc242 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:54.047066 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7a1eb06-ccc4-47c1-93e1-bd6cfe7c3a8f None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:54.114555 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a60b7aaa-9e19-44dd-ab6d-74898ddfc242 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:54.121139 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2703e66c-3fa5-43e8-99cd-c547f945d0c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:54.121794 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2703e66c-3fa5-43e8-99cd-c547f945d0c0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:54.122073 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2703e66c-3fa5-43e8-99cd-c547f945d0c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:54.122342 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2703e66c-3fa5-43e8-99cd-c547f945d0c0 admin admin] PATH_INFO: `/v3/users/b4efa5e581894a6bae36c3c4b56801c0` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:54.123772 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2703e66c-3fa5-43e8-99cd-c547f945d0c0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b4efa5e581894a6bae36c3c4b56801c0)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:54.125609 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a335ac40-337b-47f4-b390-1a05d8ef767f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:54.126514 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a335ac40-337b-47f4-b390-1a05d8ef767f service neutron] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:54.126649 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2703e66c-3fa5-43e8-99cd-c547f945d0c0 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:54.127280 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a335ac40-337b-47f4-b390-1a05d8ef767f service neutron] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:54.127757 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a335ac40-337b-47f4-b390-1a05d8ef767f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:54.130703 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-a335ac40-337b-47f4-b390-1a05d8ef767f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:54.151535 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a335ac40-337b-47f4-b390-1a05d8ef767f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:54.152858 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a335ac40-337b-47f4-b390-1a05d8ef767f service neutron] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:54.165753 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 904/1811] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:01:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 252 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 20 09:01:54.211906 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-2703e66c-3fa5-43e8-99cd-c547f945d0c0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b4efa5e581894a6bae36c3c4b56801c0'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:54.224329 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-2703e66c-3fa5-43e8-99cd-c547f945d0c0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b4efa5e581894a6bae36c3c4b56801c0'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:54.240664 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-2703e66c-3fa5-43e8-99cd-c547f945d0c0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b4efa5e581894a6bae36c3c4b56801c0'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:54.250263 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-2703e66c-3fa5-43e8-99cd-c547f945d0c0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b4efa5e581894a6bae36c3c4b56801c0'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:54.251355 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-2703e66c-3fa5-43e8-99cd-c547f945d0c0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b4efa5e581894a6bae36c3c4b56801c0'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:54.258842 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 908/1812] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:01:54 2022] DELETE /identity/v3/users/b4efa5e581894a6bae36c3c4b56801c0 => generated 0 bytes in 240 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:54.378000 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07c33bd-f4ba-4b1e-836b-1fbda03add8a None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:54.378298 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07c33bd-f4ba-4b1e-836b-1fbda03add8a None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:54.378724 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07c33bd-f4ba-4b1e-836b-1fbda03add8a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:54.411491 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-d07c33bd-f4ba-4b1e-836b-1fbda03add8a None None] MFA Rules not processed for user `7e0951446ad449d6b725fe6cc14df1f2`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:54.413646 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-d07c33bd-f4ba-4b1e-836b-1fbda03add8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:54.469286 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2703e66c-3fa5-43e8-99cd-c547f945d0c0 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:54.471280 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-2703e66c-3fa5-43e8-99cd-c547f945d0c0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:54.494670 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 905/1813] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:54 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:54.544169 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:54.587263 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2703e66c-3fa5-43e8-99cd-c547f945d0c0 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:54.592293 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-925af62e-45bf-4471-a863-1bebb5822eb8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:54.592933 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-925af62e-45bf-4471-a863-1bebb5822eb8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:54.593213 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-925af62e-45bf-4471-a863-1bebb5822eb8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:54.593484 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-925af62e-45bf-4471-a863-1bebb5822eb8 admin admin] PATH_INFO: `/v3/projects/5a5c6b9deac74d5f92cbd17bc0ca0cd2` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:54.595184 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-925af62e-45bf-4471-a863-1bebb5822eb8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5a5c6b9deac74d5f92cbd17bc0ca0cd2)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:54.597386 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-925af62e-45bf-4471-a863-1bebb5822eb8 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:54.655155 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-925af62e-45bf-4471-a863-1bebb5822eb8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5a5c6b9deac74d5f92cbd17bc0ca0cd2', 'request_id': 'req-925af62e-45bf-4471-a863-1bebb5822eb8'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:54.663094 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-925af62e-45bf-4471-a863-1bebb5822eb8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5a5c6b9deac74d5f92cbd17bc0ca0cd2', 'request_id': 'req-925af62e-45bf-4471-a863-1bebb5822eb8'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:54.669516 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-925af62e-45bf-4471-a863-1bebb5822eb8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5a5c6b9deac74d5f92cbd17bc0ca0cd2', 'request_id': 'req-925af62e-45bf-4471-a863-1bebb5822eb8'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:54.685022 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-925af62e-45bf-4471-a863-1bebb5822eb8 admin admin] The token cache is being invalidate because project 5a5c6b9deac74d5f92cbd17bc0ca0cd2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81561) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:01:54.685409 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-925af62e-45bf-4471-a863-1bebb5822eb8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:54.686118 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-925af62e-45bf-4471-a863-1bebb5822eb8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:54.688025 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 909/1814] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:01:54 2022] DELETE /identity/v3/projects/5a5c6b9deac74d5f92cbd17bc0ca0cd2 => generated 0 bytes in 219 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:54.692798 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c184b9d7-377c-42c3-8c53-e4a5c3166e2b None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:54.693322 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c184b9d7-377c-42c3-8c53-e4a5c3166e2b None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:54.693419 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c184b9d7-377c-42c3-8c53-e4a5c3166e2b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:54.719321 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-c184b9d7-377c-42c3-8c53-e4a5c3166e2b None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:54.720631 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-c184b9d7-377c-42c3-8c53-e4a5c3166e2b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:54.733771 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:54.796821 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 906/1815] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:54.802766 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71702cf8-d51d-452d-9a7b-621fddfe6348 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:54.803246 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71702cf8-d51d-452d-9a7b-621fddfe6348 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:54.803540 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71702cf8-d51d-452d-9a7b-621fddfe6348 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:54.841291 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-71702cf8-d51d-452d-9a7b-621fddfe6348 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:54.842637 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-71702cf8-d51d-452d-9a7b-621fddfe6348 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:54.852023 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 910/1816] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 20 09:01:54.856560 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c184b9d7-377c-42c3-8c53-e4a5c3166e2b None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:54.867519 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c184b9d7-377c-42c3-8c53-e4a5c3166e2b None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:54.872056 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e7c04ee-45b7-406a-bf65-b82edd281ec9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:54.872816 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7c04ee-45b7-406a-bf65-b82edd281ec9 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:54.873096 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7c04ee-45b7-406a-bf65-b82edd281ec9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:54.873362 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7c04ee-45b7-406a-bf65-b82edd281ec9 admin admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:54.874013 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e7c04ee-45b7-406a-bf65-b82edd281ec9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 09:01:54.874306 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e7c04ee-45b7-406a-bf65-b82edd281ec9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:54.875941 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e7c04ee-45b7-406a-bf65-b82edd281ec9 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:54.893431 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 907/1817] 10.222.0.36 () {58 vars in 1153 bytes} [Thu Oct 20 09:01:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 09:01:54.894626 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71702cf8-d51d-452d-9a7b-621fddfe6348 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:54.904227 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71702cf8-d51d-452d-9a7b-621fddfe6348 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:54.909102 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bd1b0bb-8b87-4d3c-b99f-1c6b1e415644 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:54.909777 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd1b0bb-8b87-4d3c-b99f-1c6b1e415644 admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:54.910119 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd1b0bb-8b87-4d3c-b99f-1c6b1e415644 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:54.910449 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd1b0bb-8b87-4d3c-b99f-1c6b1e415644 admin admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:54.911167 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bd1b0bb-8b87-4d3c-b99f-1c6b1e415644 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:54.914087 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bd1b0bb-8b87-4d3c-b99f-1c6b1e415644 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:54.949164 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 911/1818] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:01:54 2022] POST /identity/v3/projects => generated 371 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:01:54.953326 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e7c04ee-45b7-406a-bf65-b82edd281ec9 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:54.966639 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e7c04ee-45b7-406a-bf65-b82edd281ec9 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:55.031141 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b0b7b5e-c08b-428b-b67f-fa57d93611d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:55.032640 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b0b7b5e-c08b-428b-b67f-fa57d93611d4 admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:55.032921 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b0b7b5e-c08b-428b-b67f-fa57d93611d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:55.033186 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b0b7b5e-c08b-428b-b67f-fa57d93611d4 admin admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:55.034287 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b0b7b5e-c08b-428b-b67f-fa57d93611d4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:55.037536 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b0b7b5e-c08b-428b-b67f-fa57d93611d4 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:55.065157 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 908/1819] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:01:54 2022] POST /identity/v3/users => generated 286 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 20 09:01:55.069095 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bd1b0bb-8b87-4d3c-b99f-1c6b1e415644 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:55.081213 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bd1b0bb-8b87-4d3c-b99f-1c6b1e415644 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:55.088444 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bffbbde-4df3-4c7e-b519-980a0d54b42a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:55.089322 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bffbbde-4df3-4c7e-b519-980a0d54b42a admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:55.089665 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bffbbde-4df3-4c7e-b519-980a0d54b42a admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:55.090342 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bffbbde-4df3-4c7e-b519-980a0d54b42a admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:55.091194 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bffbbde-4df3-4c7e-b519-980a0d54b42a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:55.093685 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bffbbde-4df3-4c7e-b519-980a0d54b42a admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:55.094000 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b0b7b5e-c08b-428b-b67f-fa57d93611d4 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:55.095842 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-9b0b7b5e-c08b-428b-b67f-fa57d93611d4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:55.110504 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 912/1820] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:01:55 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 20 09:01:55.114581 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bffbbde-4df3-4c7e-b519-980a0d54b42a admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:55.119580 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b0b7b5e-c08b-428b-b67f-fa57d93611d4 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:55.127241 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bffbbde-4df3-4c7e-b519-980a0d54b42a admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:55.128336 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-326b683c-a076-45b2-b0bd-e5af61179bd7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:55.129162 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326b683c-a076-45b2-b0bd-e5af61179bd7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:55.129445 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326b683c-a076-45b2-b0bd-e5af61179bd7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:55.129716 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326b683c-a076-45b2-b0bd-e5af61179bd7 admin admin] PATH_INFO: `/v3/projects/b4e516e9d13841a480ccd13a4ae5acd0` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:55.130902 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-326b683c-a076-45b2-b0bd-e5af61179bd7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b4e516e9d13841a480ccd13a4ae5acd0)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:55.133481 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-326b683c-a076-45b2-b0bd-e5af61179bd7 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:55.133674 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf76f92d-c1ad-47ff-abae-25be2b3c0649 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:55.134602 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf76f92d-c1ad-47ff-abae-25be2b3c0649 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:55.135229 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf76f92d-c1ad-47ff-abae-25be2b3c0649 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:55.135492 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf76f92d-c1ad-47ff-abae-25be2b3c0649 admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:55.136962 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf76f92d-c1ad-47ff-abae-25be2b3c0649 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:55.139188 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf76f92d-c1ad-47ff-abae-25be2b3c0649 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:55.153297 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 913/1821] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:01:55 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 20 09:01:55.158107 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf76f92d-c1ad-47ff-abae-25be2b3c0649 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:55.170019 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf76f92d-c1ad-47ff-abae-25be2b3c0649 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:55.175472 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3812a1c-d506-4a70-b5e3-cf5f891e569a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:55.176273 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3812a1c-d506-4a70-b5e3-cf5f891e569a admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:55.176662 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3812a1c-d506-4a70-b5e3-cf5f891e569a admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:55.176900 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3812a1c-d506-4a70-b5e3-cf5f891e569a admin admin] PATH_INFO: `/v3/projects/9e0899a46ea34a93be8001366fccac4e/users/5d408c12fb2142bea043c68221b2bebe/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:55.196175 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3812a1c-d506-4a70-b5e3-cf5f891e569a admin admin] RBAC: Authorizing `identity:create_grant(project_id=9e0899a46ea34a93be8001366fccac4e, user_id=5d408c12fb2142bea043c68221b2bebe, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:55.198767 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-326b683c-a076-45b2-b0bd-e5af61179bd7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b4e516e9d13841a480ccd13a4ae5acd0', 'request_id': 'req-326b683c-a076-45b2-b0bd-e5af61179bd7'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:55.200149 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3812a1c-d506-4a70-b5e3-cf5f891e569a admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:55.210183 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-326b683c-a076-45b2-b0bd-e5af61179bd7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b4e516e9d13841a480ccd13a4ae5acd0', 'request_id': 'req-326b683c-a076-45b2-b0bd-e5af61179bd7'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:55.221116 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-326b683c-a076-45b2-b0bd-e5af61179bd7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b4e516e9d13841a480ccd13a4ae5acd0', 'request_id': 'req-326b683c-a076-45b2-b0bd-e5af61179bd7'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:55.226578 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 914/1822] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:01:55 2022] PUT /identity/v3/projects/9e0899a46ea34a93be8001366fccac4e/users/5d408c12fb2142bea043c68221b2bebe/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:55.231404 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d20dfd-d921-4ac6-bf20-f7e54a9856a7 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:55.231704 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d20dfd-d921-4ac6-bf20-f7e54a9856a7 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:55.232120 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d20dfd-d921-4ac6-bf20-f7e54a9856a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:55.245184 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-326b683c-a076-45b2-b0bd-e5af61179bd7 admin admin] The token cache is being invalidate because project b4e516e9d13841a480ccd13a4ae5acd0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81562) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:01:55.245602 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-326b683c-a076-45b2-b0bd-e5af61179bd7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:55.247012 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-326b683c-a076-45b2-b0bd-e5af61179bd7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:55.252260 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 909/1823] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:01:55 2022] DELETE /identity/v3/projects/b4e516e9d13841a480ccd13a4ae5acd0 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:55.253305 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-326b683c-a076-45b2-b0bd-e5af61179bd7 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:55.255030 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-326b683c-a076-45b2-b0bd-e5af61179bd7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:55.280717 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-46d20dfd-d921-4ac6-bf20-f7e54a9856a7 None None] MFA Rules not processed for user `5d408c12fb2142bea043c68221b2bebe`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:55.282180 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-46d20dfd-d921-4ac6-bf20-f7e54a9856a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:55.300219 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:55.366149 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-326b683c-a076-45b2-b0bd-e5af61179bd7 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:55.397145 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:55.449418 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 915/1824] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:55 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 219 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 20 09:01:55.461965 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46d20dfd-d921-4ac6-bf20-f7e54a9856a7 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:55.463892 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-46d20dfd-d921-4ac6-bf20-f7e54a9856a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:55.471850 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d91de44-01e9-4fcc-b27e-2375562310ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:55.473564 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d91de44-01e9-4fcc-b27e-2375562310ad admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:55.474195 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d91de44-01e9-4fcc-b27e-2375562310ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:55.474526 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d91de44-01e9-4fcc-b27e-2375562310ad admin admin] PATH_INFO: `/v3/users/bf7195dce66243f488b833e2197db2a6` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:55.479133 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d91de44-01e9-4fcc-b27e-2375562310ad admin admin] RBAC: Authorizing `identity:delete_user(user_id=bf7195dce66243f488b833e2197db2a6)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:55.481112 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d91de44-01e9-4fcc-b27e-2375562310ad admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:55.554287 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-5d91de44-01e9-4fcc-b27e-2375562310ad admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bf7195dce66243f488b833e2197db2a6'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:55.568654 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-5d91de44-01e9-4fcc-b27e-2375562310ad admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bf7195dce66243f488b833e2197db2a6'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:55.586929 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46d20dfd-d921-4ac6-bf20-f7e54a9856a7 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:55.586929 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fee1d9cb-b3e6-4f92-ad72-512d36a507e6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:55.586929 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee1d9cb-b3e6-4f92-ad72-512d36a507e6 service cinder] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:55.586929 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee1d9cb-b3e6-4f92-ad72-512d36a507e6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:55.587254 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee1d9cb-b3e6-4f92-ad72-512d36a507e6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:55.587337 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-5d91de44-01e9-4fcc-b27e-2375562310ad admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bf7195dce66243f488b833e2197db2a6'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:55.594373 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-5d91de44-01e9-4fcc-b27e-2375562310ad admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bf7195dce66243f488b833e2197db2a6'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:55.595295 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-5d91de44-01e9-4fcc-b27e-2375562310ad admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bf7195dce66243f488b833e2197db2a6'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:55.595451 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fee1d9cb-b3e6-4f92-ad72-512d36a507e6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:55.597915 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fee1d9cb-b3e6-4f92-ad72-512d36a507e6 service cinder] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:55.605165 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 910/1825] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:01:55 2022] DELETE /identity/v3/users/bf7195dce66243f488b833e2197db2a6 => generated 0 bytes in 352 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:55.622635 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 916/1826] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:01:55 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 160 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:01:55.636898 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d91de44-01e9-4fcc-b27e-2375562310ad admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:55.638715 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-5d91de44-01e9-4fcc-b27e-2375562310ad admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:55.730917 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:55.764952 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d91de44-01e9-4fcc-b27e-2375562310ad admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:55.782176 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feac7ae2-6621-4323-a6d2-44560fae78cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:55.782901 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feac7ae2-6621-4323-a6d2-44560fae78cc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:55.783246 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feac7ae2-6621-4323-a6d2-44560fae78cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:55.783527 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feac7ae2-6621-4323-a6d2-44560fae78cc admin admin] PATH_INFO: `/v3/users/5d408c12fb2142bea043c68221b2bebe` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:55.784687 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feac7ae2-6621-4323-a6d2-44560fae78cc admin admin] RBAC: Authorizing `identity:delete_user(user_id=5d408c12fb2142bea043c68221b2bebe)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:55.786910 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feac7ae2-6621-4323-a6d2-44560fae78cc admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:55.850443 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-feac7ae2-6621-4323-a6d2-44560fae78cc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5d408c12fb2142bea043c68221b2bebe'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:55.861481 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-feac7ae2-6621-4323-a6d2-44560fae78cc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5d408c12fb2142bea043c68221b2bebe'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:55.878891 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-feac7ae2-6621-4323-a6d2-44560fae78cc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5d408c12fb2142bea043c68221b2bebe'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:55.891059 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-feac7ae2-6621-4323-a6d2-44560fae78cc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5d408c12fb2142bea043c68221b2bebe'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:55.891509 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-feac7ae2-6621-4323-a6d2-44560fae78cc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5d408c12fb2142bea043c68221b2bebe'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:55.900127 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 911/1827] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:01:55 2022] DELETE /identity/v3/users/5d408c12fb2142bea043c68221b2bebe => generated 0 bytes in 264 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:55.904645 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dce3850-855a-4b56-9b6d-521378bb5e5a None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:55.905131 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dce3850-855a-4b56-9b6d-521378bb5e5a None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:55.905225 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dce3850-855a-4b56-9b6d-521378bb5e5a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:55.950519 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-6dce3850-855a-4b56-9b6d-521378bb5e5a None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:55.961214 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-6dce3850-855a-4b56-9b6d-521378bb5e5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:56.003227 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feac7ae2-6621-4323-a6d2-44560fae78cc admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:56.005438 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-feac7ae2-6621-4323-a6d2-44560fae78cc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:56.078851 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 917/1828] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:56.088561 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dce3850-855a-4b56-9b6d-521378bb5e5a None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:56.092032 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-6dce3850-855a-4b56-9b6d-521378bb5e5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:56.145902 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feac7ae2-6621-4323-a6d2-44560fae78cc admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:56.158891 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1207fbb-d54d-482b-b787-a4362b0e2f56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:56.159549 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1207fbb-d54d-482b-b787-a4362b0e2f56 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:56.159828 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1207fbb-d54d-482b-b787-a4362b0e2f56 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:56.160186 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1207fbb-d54d-482b-b787-a4362b0e2f56 admin admin] PATH_INFO: `/v3/projects/56353c9fefe44e2d99ea268d5d05e314` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:56.164726 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1207fbb-d54d-482b-b787-a4362b0e2f56 admin admin] RBAC: Authorizing `identity:delete_project(project_id=56353c9fefe44e2d99ea268d5d05e314)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:56.169690 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1207fbb-d54d-482b-b787-a4362b0e2f56 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:56.188299 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dce3850-855a-4b56-9b6d-521378bb5e5a None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:56.197337 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8a4ecbe-ce08-4b59-89eb-31c1b4dd2180 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:56.198210 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a4ecbe-ce08-4b59-89eb-31c1b4dd2180 service neutron] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:56.198710 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a4ecbe-ce08-4b59-89eb-31c1b4dd2180 service neutron] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:56.200046 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a4ecbe-ce08-4b59-89eb-31c1b4dd2180 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:56.217226 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8a4ecbe-ce08-4b59-89eb-31c1b4dd2180 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:56.223134 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8a4ecbe-ce08-4b59-89eb-31c1b4dd2180 service neutron] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:56.238170 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 918/1829] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:01:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 150 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:01:56.276562 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-b1207fbb-d54d-482b-b787-a4362b0e2f56 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '56353c9fefe44e2d99ea268d5d05e314', 'request_id': 'req-b1207fbb-d54d-482b-b787-a4362b0e2f56'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:56.286338 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-b1207fbb-d54d-482b-b787-a4362b0e2f56 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '56353c9fefe44e2d99ea268d5d05e314', 'request_id': 'req-b1207fbb-d54d-482b-b787-a4362b0e2f56'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:56.293549 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-b1207fbb-d54d-482b-b787-a4362b0e2f56 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '56353c9fefe44e2d99ea268d5d05e314', 'request_id': 'req-b1207fbb-d54d-482b-b787-a4362b0e2f56'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:56.296060 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=83,keystone:DELETE=16,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:56.323030 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-b1207fbb-d54d-482b-b787-a4362b0e2f56 admin admin] The token cache is being invalidate because project 56353c9fefe44e2d99ea268d5d05e314 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81562) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:01:56.323369 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-b1207fbb-d54d-482b-b787-a4362b0e2f56 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:56.324078 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-b1207fbb-d54d-482b-b787-a4362b0e2f56 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:56.325959 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 912/1830] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:01:56 2022] DELETE /identity/v3/projects/56353c9fefe44e2d99ea268d5d05e314 => generated 0 bytes in 323 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:56.332795 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa12783d-af16-492d-bdfc-f05c1e2eeb0c None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:56.333326 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa12783d-af16-492d-bdfc-f05c1e2eeb0c None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:56.333618 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa12783d-af16-492d-bdfc-f05c1e2eeb0c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:56.361051 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-fa12783d-af16-492d-bdfc-f05c1e2eeb0c None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:56.362255 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-fa12783d-af16-492d-bdfc-f05c1e2eeb0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:56.392936 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:56.441292 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 919/1831] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 20 09:01:56.451811 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c0ca8e-ae99-4e7a-b95a-617af17ccc07 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:56.452104 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c0ca8e-ae99-4e7a-b95a-617af17ccc07 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:56.453289 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c0ca8e-ae99-4e7a-b95a-617af17ccc07 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:56.480461 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-e1c0ca8e-ae99-4e7a-b95a-617af17ccc07 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:56.481987 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-e1c0ca8e-ae99-4e7a-b95a-617af17ccc07 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:56.496826 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 913/1832] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:56.497707 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa12783d-af16-492d-bdfc-f05c1e2eeb0c None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:56.511948 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa12783d-af16-492d-bdfc-f05c1e2eeb0c None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:56.517200 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a12e62a9-6ec5-4509-ad82-ec0c7fa27025 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:56.517819 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12e62a9-6ec5-4509-ad82-ec0c7fa27025 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:56.518092 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12e62a9-6ec5-4509-ad82-ec0c7fa27025 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:56.518367 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12e62a9-6ec5-4509-ad82-ec0c7fa27025 admin admin] PATH_INFO: `/v3/domains` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:56.519168 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a12e62a9-6ec5-4509-ad82-ec0c7fa27025 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 09:01:56.519475 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a12e62a9-6ec5-4509-ad82-ec0c7fa27025 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:56.521147 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a12e62a9-6ec5-4509-ad82-ec0c7fa27025 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:56.538063 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 920/1833] 10.222.0.36 () {58 vars in 1153 bytes} [Thu Oct 20 09:01:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 09:01:56.540866 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1c0ca8e-ae99-4e7a-b95a-617af17ccc07 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:56.555332 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1c0ca8e-ae99-4e7a-b95a-617af17ccc07 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:56.560745 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eec528d-4415-4637-be80-0dd3694101f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:56.562032 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eec528d-4415-4637-be80-0dd3694101f4 admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:56.562646 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eec528d-4415-4637-be80-0dd3694101f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:56.563142 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eec528d-4415-4637-be80-0dd3694101f4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:56.564520 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eec528d-4415-4637-be80-0dd3694101f4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:56.567199 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eec528d-4415-4637-be80-0dd3694101f4 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:56.624483 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 914/1834] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:01:56 2022] POST /identity/v3/projects => generated 377 bytes in 84 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:01:56.628970 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a12e62a9-6ec5-4509-ad82-ec0c7fa27025 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:56.650356 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a12e62a9-6ec5-4509-ad82-ec0c7fa27025 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:56.777372 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eec528d-4415-4637-be80-0dd3694101f4 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:56.779738 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-3eec528d-4415-4637-be80-0dd3694101f4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:56.792666 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-751387b8-de75-4abf-bb11-c67bf547697e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:56.793301 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-751387b8-de75-4abf-bb11-c67bf547697e admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:56.793578 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-751387b8-de75-4abf-bb11-c67bf547697e admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:56.793850 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-751387b8-de75-4abf-bb11-c67bf547697e admin admin] PATH_INFO: `/v3/users` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:56.794496 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-751387b8-de75-4abf-bb11-c67bf547697e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:56.796449 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-751387b8-de75-4abf-bb11-c67bf547697e admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:56.841063 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 921/1835] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:01:56 2022] POST /identity/v3/users => generated 289 bytes in 213 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:01:56.845275 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-751387b8-de75-4abf-bb11-c67bf547697e admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:56.871655 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-751387b8-de75-4abf-bb11-c67bf547697e admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:56.882419 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f737d0b1-dc3a-41d2-a0c5-d5d5161b58d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:56.883070 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f737d0b1-dc3a-41d2-a0c5-d5d5161b58d8 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:56.883348 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f737d0b1-dc3a-41d2-a0c5-d5d5161b58d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:56.883612 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f737d0b1-dc3a-41d2-a0c5-d5d5161b58d8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:56.884264 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f737d0b1-dc3a-41d2-a0c5-d5d5161b58d8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:56.885888 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f737d0b1-dc3a-41d2-a0c5-d5d5161b58d8 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:56.920490 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 922/1836] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:01:56 2022] GET /identity/v3/roles => generated 1420 bytes in 76 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 20 09:01:56.926615 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f737d0b1-dc3a-41d2-a0c5-d5d5161b58d8 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:56.930381 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eec528d-4415-4637-be80-0dd3694101f4 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:56.938625 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f737d0b1-dc3a-41d2-a0c5-d5d5161b58d8 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:56.942654 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd495091-419b-4307-a1ef-434ef624a430 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:56.943388 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd495091-419b-4307-a1ef-434ef624a430 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:56.943736 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd495091-419b-4307-a1ef-434ef624a430 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:56.944083 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd495091-419b-4307-a1ef-434ef624a430 admin admin] PATH_INFO: `/v3/projects/9e0899a46ea34a93be8001366fccac4e` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:56.945328 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98578698-4de1-46eb-920a-3ccb456cacb4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:56.946106 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd495091-419b-4307-a1ef-434ef624a430 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9e0899a46ea34a93be8001366fccac4e)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:56.946358 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98578698-4de1-46eb-920a-3ccb456cacb4 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:56.946846 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98578698-4de1-46eb-920a-3ccb456cacb4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:56.947311 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98578698-4de1-46eb-920a-3ccb456cacb4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:56.948163 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98578698-4de1-46eb-920a-3ccb456cacb4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:56.949848 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98578698-4de1-46eb-920a-3ccb456cacb4 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:56.951100 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd495091-419b-4307-a1ef-434ef624a430 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:56.962156 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 923/1837] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:01:56 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:01:56.966796 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98578698-4de1-46eb-920a-3ccb456cacb4 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:56.979525 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98578698-4de1-46eb-920a-3ccb456cacb4 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:56.984777 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-909902e1-38c1-49ef-a414-feffb5fa8334 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:56.985633 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-909902e1-38c1-49ef-a414-feffb5fa8334 admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:56.986063 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-909902e1-38c1-49ef-a414-feffb5fa8334 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:56.986680 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-909902e1-38c1-49ef-a414-feffb5fa8334 admin admin] PATH_INFO: `/v3/projects/f5bb1132fe1e4780946808c8b7c59479/users/a6e64de6ce4c484fb461ec4d0d8a60a3/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:57.013449 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-909902e1-38c1-49ef-a414-feffb5fa8334 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f5bb1132fe1e4780946808c8b7c59479, user_id=a6e64de6ce4c484fb461ec4d0d8a60a3, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:57.017988 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-909902e1-38c1-49ef-a414-feffb5fa8334 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:57.052684 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 924/1838] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:01:56 2022] PUT /identity/v3/projects/f5bb1132fe1e4780946808c8b7c59479/users/a6e64de6ce4c484fb461ec4d0d8a60a3/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 86 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:57.064414 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab4e7888-0b79-4b00-bb4c-f4baa481f9f2 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:57.064752 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab4e7888-0b79-4b00-bb4c-f4baa481f9f2 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:57.065025 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab4e7888-0b79-4b00-bb4c-f4baa481f9f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:57.081551 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-bd495091-419b-4307-a1ef-434ef624a430 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9e0899a46ea34a93be8001366fccac4e', 'request_id': 'req-bd495091-419b-4307-a1ef-434ef624a430'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:57.089313 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-bd495091-419b-4307-a1ef-434ef624a430 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9e0899a46ea34a93be8001366fccac4e', 'request_id': 'req-bd495091-419b-4307-a1ef-434ef624a430'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:57.096977 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-bd495091-419b-4307-a1ef-434ef624a430 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9e0899a46ea34a93be8001366fccac4e', 'request_id': 'req-bd495091-419b-4307-a1ef-434ef624a430'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:57.131341 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-bd495091-419b-4307-a1ef-434ef624a430 admin admin] The token cache is being invalidate because project 9e0899a46ea34a93be8001366fccac4e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81562) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:01:57.131652 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-bd495091-419b-4307-a1ef-434ef624a430 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:57.132330 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-ab4e7888-0b79-4b00-bb4c-f4baa481f9f2 None None] MFA Rules not processed for user `a6e64de6ce4c484fb461ec4d0d8a60a3`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:57.133810 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-bd495091-419b-4307-a1ef-434ef624a430 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:57.135752 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 915/1839] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:01:56 2022] DELETE /identity/v3/projects/9e0899a46ea34a93be8001366fccac4e => generated 0 bytes in 359 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:57.142918 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae3413d-b3be-4972-8653-92b9ea0ac922 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:57.143292 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae3413d-b3be-4972-8653-92b9ea0ac922 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:57.143628 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae3413d-b3be-4972-8653-92b9ea0ac922 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:57.143776 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-ab4e7888-0b79-4b00-bb4c-f4baa481f9f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:57.176646 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-4ae3413d-b3be-4972-8653-92b9ea0ac922 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:57.180384 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-4ae3413d-b3be-4972-8653-92b9ea0ac922 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:57.257665 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:INSERT=3,keystone:DELETE=8,keystone:SELECT=89 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:57.286414 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:57.303700 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 916/1840] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:57.311616 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76092799-a2c4-432e-be4d-3871e1db63ce None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:57.312330 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76092799-a2c4-432e-be4d-3871e1db63ce None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:57.312757 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76092799-a2c4-432e-be4d-3871e1db63ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:57.349033 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 925/1841] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:57 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 285 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:57.350839 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab4e7888-0b79-4b00-bb4c-f4baa481f9f2 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:57.354446 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-ab4e7888-0b79-4b00-bb4c-f4baa481f9f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:57.366536 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-76092799-a2c4-432e-be4d-3871e1db63ce None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:57.366536 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-76092799-a2c4-432e-be4d-3871e1db63ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:57.392263 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 917/1842] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 20 09:01:57.396902 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76092799-a2c4-432e-be4d-3871e1db63ce None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:57.426975 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76092799-a2c4-432e-be4d-3871e1db63ce None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:57.447177 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca39c42-4278-46d2-94c7-3950e6a513e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:57.447798 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca39c42-4278-46d2-94c7-3950e6a513e6 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:57.448076 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca39c42-4278-46d2-94c7-3950e6a513e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:57.448358 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca39c42-4278-46d2-94c7-3950e6a513e6 admin admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:57.449071 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cca39c42-4278-46d2-94c7-3950e6a513e6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 09:01:57.449447 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cca39c42-4278-46d2-94c7-3950e6a513e6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:57.451182 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cca39c42-4278-46d2-94c7-3950e6a513e6 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:57.481710 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 918/1843] 10.222.0.36 () {58 vars in 1153 bytes} [Thu Oct 20 09:01:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 09:01:57.482810 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca39c42-4278-46d2-94c7-3950e6a513e6 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:57.495043 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca39c42-4278-46d2-94c7-3950e6a513e6 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:57.502881 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6245cf2-f397-46a6-9e2e-99ae9f5064a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:57.503586 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6245cf2-f397-46a6-9e2e-99ae9f5064a7 admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:57.503936 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6245cf2-f397-46a6-9e2e-99ae9f5064a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:57.504313 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6245cf2-f397-46a6-9e2e-99ae9f5064a7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:57.505049 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6245cf2-f397-46a6-9e2e-99ae9f5064a7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:57.507878 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6245cf2-f397-46a6-9e2e-99ae9f5064a7 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:57.508821 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab4e7888-0b79-4b00-bb4c-f4baa481f9f2 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:57.539667 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95067923-bf90-4f41-b5fa-8d8e331bfedb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:57.540215 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95067923-bf90-4f41-b5fa-8d8e331bfedb service cinder] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:57.540489 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95067923-bf90-4f41-b5fa-8d8e331bfedb service cinder] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:57.540750 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95067923-bf90-4f41-b5fa-8d8e331bfedb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:57.565430 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95067923-bf90-4f41-b5fa-8d8e331bfedb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:57.566928 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95067923-bf90-4f41-b5fa-8d8e331bfedb service cinder] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:57.592189 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 919/1844] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:01:57 2022] POST /identity/v3/projects => generated 359 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:01:57.597830 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6245cf2-f397-46a6-9e2e-99ae9f5064a7 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:57.620051 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 926/1845] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:01:57 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 270 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:01:57.748419 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6245cf2-f397-46a6-9e2e-99ae9f5064a7 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:57.760290 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d372055b-33e5-4af0-90ef-a1303a936d20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:57.760924 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d372055b-33e5-4af0-90ef-a1303a936d20 admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:57.761206 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d372055b-33e5-4af0-90ef-a1303a936d20 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:57.761592 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d372055b-33e5-4af0-90ef-a1303a936d20 admin admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:57.762515 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d372055b-33e5-4af0-90ef-a1303a936d20 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:57.764114 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d372055b-33e5-4af0-90ef-a1303a936d20 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:57.797354 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 920/1846] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:01:57 2022] POST /identity/v3/users => generated 280 bytes in 204 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:01:57.802570 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95067923-bf90-4f41-b5fa-8d8e331bfedb service cinder] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:57.830052 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95067923-bf90-4f41-b5fa-8d8e331bfedb service cinder] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:57.849207 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a20983-5bed-4583-b883-3c2232bade7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:57.849907 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a20983-5bed-4583-b883-3c2232bade7a admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:57.850303 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a20983-5bed-4583-b883-3c2232bade7a admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:57.850697 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a20983-5bed-4583-b883-3c2232bade7a admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:57.851469 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4a20983-5bed-4583-b883-3c2232bade7a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:57.854736 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4a20983-5bed-4583-b883-3c2232bade7a admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:57.872846 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 927/1847] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:01:57 2022] GET /identity/v3/roles => generated 1420 bytes in 71 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:01:57.877305 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d372055b-33e5-4af0-90ef-a1303a936d20 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:57.893546 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d372055b-33e5-4af0-90ef-a1303a936d20 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:57.909665 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-086ba95d-4eac-48b6-be12-4a856d02ac6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:57.910664 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-086ba95d-4eac-48b6-be12-4a856d02ac6d admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:57.911521 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-086ba95d-4eac-48b6-be12-4a856d02ac6d admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:57.912197 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-086ba95d-4eac-48b6-be12-4a856d02ac6d admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:57.913039 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-086ba95d-4eac-48b6-be12-4a856d02ac6d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:57.918459 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-086ba95d-4eac-48b6-be12-4a856d02ac6d admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:57.962685 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 921/1848] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:01:57 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:01:57.963077 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a20983-5bed-4583-b883-3c2232bade7a admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:57.980659 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a20983-5bed-4583-b883-3c2232bade7a admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:57.996339 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ca9eb78-e9e7-4557-b21d-01005803a4f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:57.998445 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca9eb78-e9e7-4557-b21d-01005803a4f7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:57.999085 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca9eb78-e9e7-4557-b21d-01005803a4f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:58.000828 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca9eb78-e9e7-4557-b21d-01005803a4f7 admin admin] PATH_INFO: `/v3/projects/f609577d69a54fc38215e32ef577a3f4/users/8803b1f168594b75b1271a65065d9b33/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:58.027386 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ca9eb78-e9e7-4557-b21d-01005803a4f7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f609577d69a54fc38215e32ef577a3f4, user_id=8803b1f168594b75b1271a65065d9b33, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:58.037088 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ca9eb78-e9e7-4557-b21d-01005803a4f7 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:58.089591 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 928/1849] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:01:57 2022] PUT /identity/v3/projects/f609577d69a54fc38215e32ef577a3f4/users/8803b1f168594b75b1271a65065d9b33/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 127 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:01:58.105244 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2afb52-7dfe-42da-bcee-f4a16be9f284 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:58.105525 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2afb52-7dfe-42da-bcee-f4a16be9f284 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:58.105796 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2afb52-7dfe-42da-bcee-f4a16be9f284 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:58.162878 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-ff2afb52-7dfe-42da-bcee-f4a16be9f284 None None] MFA Rules not processed for user `8803b1f168594b75b1271a65065d9b33`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:58.164995 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-ff2afb52-7dfe-42da-bcee-f4a16be9f284 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:58.285543 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 922/1850] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:58 2022] POST /identity/v3/auth/tokens => generated 3236 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:58.331341 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ca9eb78-e9e7-4557-b21d-01005803a4f7 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:58.407722 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ca9eb78-e9e7-4557-b21d-01005803a4f7 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:58.412619 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14dc2d85-f486-4715-8a29-6bef702946e2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:58.414025 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14dc2d85-f486-4715-8a29-6bef702946e2 service cinder] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:58.415663 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14dc2d85-f486-4715-8a29-6bef702946e2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:58.416115 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14dc2d85-f486-4715-8a29-6bef702946e2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:58.433891 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14dc2d85-f486-4715-8a29-6bef702946e2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:58.435527 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14dc2d85-f486-4715-8a29-6bef702946e2 service cinder] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:58.448009 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 929/1851] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:01:58 2022] GET /identity/v3/auth/tokens => generated 3236 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:01:58.465797 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff2afb52-7dfe-42da-bcee-f4a16be9f284 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:58.488889 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:58.538952 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff2afb52-7dfe-42da-bcee-f4a16be9f284 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:58.545002 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3278d91-3d68-4789-8149-2ee4d496b8cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:58.545679 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3278d91-3d68-4789-8149-2ee4d496b8cb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:58.546034 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3278d91-3d68-4789-8149-2ee4d496b8cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:58.546376 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3278d91-3d68-4789-8149-2ee4d496b8cb admin admin] PATH_INFO: `/v3/users/8803b1f168594b75b1271a65065d9b33` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:58.547679 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3278d91-3d68-4789-8149-2ee4d496b8cb admin admin] RBAC: Authorizing `identity:delete_user(user_id=8803b1f168594b75b1271a65065d9b33)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:58.549861 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3278d91-3d68-4789-8149-2ee4d496b8cb admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:58.602955 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-c3278d91-3d68-4789-8149-2ee4d496b8cb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8803b1f168594b75b1271a65065d9b33'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:58.611102 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-c3278d91-3d68-4789-8149-2ee4d496b8cb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8803b1f168594b75b1271a65065d9b33'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:58.624544 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-c3278d91-3d68-4789-8149-2ee4d496b8cb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8803b1f168594b75b1271a65065d9b33'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:58.632775 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-c3278d91-3d68-4789-8149-2ee4d496b8cb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8803b1f168594b75b1271a65065d9b33'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:58.633726 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-c3278d91-3d68-4789-8149-2ee4d496b8cb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8803b1f168594b75b1271a65065d9b33'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:58.640513 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 923/1852] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:01:58 2022] DELETE /identity/v3/users/8803b1f168594b75b1271a65065d9b33 => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:01:58.646236 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1fa2c45-81d3-4f02-98d8-c74ba70c32b1 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:58.646789 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1fa2c45-81d3-4f02-98d8-c74ba70c32b1 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:58.647130 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1fa2c45-81d3-4f02-98d8-c74ba70c32b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:58.673732 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-d1fa2c45-81d3-4f02-98d8-c74ba70c32b1 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:01:58.675087 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-d1fa2c45-81d3-4f02-98d8-c74ba70c32b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:58.771509 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:01:58.789577 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 930/1853] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:01:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:01:58.801823 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3278d91-3d68-4789-8149-2ee4d496b8cb admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:58.804325 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-c3278d91-3d68-4789-8149-2ee4d496b8cb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:58.959391 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3278d91-3d68-4789-8149-2ee4d496b8cb admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:58.963804 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-463b981f-cd8f-4f25-803f-aa337bf90618 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:58.964727 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-463b981f-cd8f-4f25-803f-aa337bf90618 service neutron] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:58.965615 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-463b981f-cd8f-4f25-803f-aa337bf90618 service neutron] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:58.966120 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-463b981f-cd8f-4f25-803f-aa337bf90618 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:58.978639 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-463b981f-cd8f-4f25-803f-aa337bf90618 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:58.980268 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-463b981f-cd8f-4f25-803f-aa337bf90618 service neutron] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:59.002707 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 924/1854] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:01:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 201 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:01:59.744514 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1fa2c45-81d3-4f02-98d8-c74ba70c32b1 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:01:59.746058 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-d1fa2c45-81d3-4f02-98d8-c74ba70c32b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:01:59.758101 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1fa2c45-81d3-4f02-98d8-c74ba70c32b1 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:01:59.762732 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5e3027d-9d92-44e4-b392-3aaac4f83420 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:01:59.763434 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e3027d-9d92-44e4-b392-3aaac4f83420 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:01:59.763794 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e3027d-9d92-44e4-b392-3aaac4f83420 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:01:59.764132 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e3027d-9d92-44e4-b392-3aaac4f83420 admin admin] PATH_INFO: `/v3/projects/f609577d69a54fc38215e32ef577a3f4` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:01:59.765309 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5e3027d-9d92-44e4-b392-3aaac4f83420 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f609577d69a54fc38215e32ef577a3f4)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:01:59.767335 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5e3027d-9d92-44e4-b392-3aaac4f83420 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:01:59.824378 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-a5e3027d-9d92-44e4-b392-3aaac4f83420 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f609577d69a54fc38215e32ef577a3f4', 'request_id': 'req-a5e3027d-9d92-44e4-b392-3aaac4f83420'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:59.834881 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-a5e3027d-9d92-44e4-b392-3aaac4f83420 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f609577d69a54fc38215e32ef577a3f4', 'request_id': 'req-a5e3027d-9d92-44e4-b392-3aaac4f83420'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:59.842153 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-a5e3027d-9d92-44e4-b392-3aaac4f83420 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f609577d69a54fc38215e32ef577a3f4', 'request_id': 'req-a5e3027d-9d92-44e4-b392-3aaac4f83420'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:59.858274 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-a5e3027d-9d92-44e4-b392-3aaac4f83420 admin admin] The token cache is being invalidate because project f609577d69a54fc38215e32ef577a3f4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81561) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:01:59.858681 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-a5e3027d-9d92-44e4-b392-3aaac4f83420 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:59.859393 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-a5e3027d-9d92-44e4-b392-3aaac4f83420 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:01:59.861641 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 931/1855] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:01:59 2022] DELETE /identity/v3/projects/f609577d69a54fc38215e32ef577a3f4 => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:02:00.091937 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-463b981f-cd8f-4f25-803f-aa337bf90618 service neutron] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:02:00.093408 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-463b981f-cd8f-4f25-803f-aa337bf90618 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:02:00.113975 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-463b981f-cd8f-4f25-803f-aa337bf90618 service neutron] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:02:00.176394 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:02:00.194949 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85891d9d-7b35-4224-a849-eda534f204fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:02:00.195578 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85891d9d-7b35-4224-a849-eda534f204fa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:02:00.195851 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85891d9d-7b35-4224-a849-eda534f204fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:02:00.196117 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85891d9d-7b35-4224-a849-eda534f204fa admin admin] PATH_INFO: `/v3/users/c7c40a4592874732871eaabd04d8662c` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:02:00.197357 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85891d9d-7b35-4224-a849-eda534f204fa admin admin] RBAC: Authorizing `identity:delete_user(user_id=c7c40a4592874732871eaabd04d8662c)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:02:00.199330 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85891d9d-7b35-4224-a849-eda534f204fa admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:02:00.252622 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-85891d9d-7b35-4224-a849-eda534f204fa admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c7c40a4592874732871eaabd04d8662c'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:02:00.260689 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-85891d9d-7b35-4224-a849-eda534f204fa admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c7c40a4592874732871eaabd04d8662c'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:02:00.273838 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-85891d9d-7b35-4224-a849-eda534f204fa admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c7c40a4592874732871eaabd04d8662c'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:02:00.280897 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-85891d9d-7b35-4224-a849-eda534f204fa admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c7c40a4592874732871eaabd04d8662c'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:02:00.281642 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-85891d9d-7b35-4224-a849-eda534f204fa admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c7c40a4592874732871eaabd04d8662c'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:02:00.288087 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 925/1856] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:02:00 2022] DELETE /identity/v3/users/c7c40a4592874732871eaabd04d8662c => generated 0 bytes in 197 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:02:00.466858 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5e3027d-9d92-44e4-b392-3aaac4f83420 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:02:00.468563 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-a5e3027d-9d92-44e4-b392-3aaac4f83420 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:02:00.482812 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5e3027d-9d92-44e4-b392-3aaac4f83420 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:02:00.545931 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7479c04b-b1f2-47a1-a81c-96a35bfff901 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:02:00.546594 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7479c04b-b1f2-47a1-a81c-96a35bfff901 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:02:00.546878 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7479c04b-b1f2-47a1-a81c-96a35bfff901 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:02:00.547150 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7479c04b-b1f2-47a1-a81c-96a35bfff901 admin admin] PATH_INFO: `/v3/projects/4da29c9977ca44baa867b3c7fc86164b` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:02:00.548289 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7479c04b-b1f2-47a1-a81c-96a35bfff901 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4da29c9977ca44baa867b3c7fc86164b)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:02:00.550204 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7479c04b-b1f2-47a1-a81c-96a35bfff901 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:02:00.595943 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-7479c04b-b1f2-47a1-a81c-96a35bfff901 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4da29c9977ca44baa867b3c7fc86164b', 'request_id': 'req-7479c04b-b1f2-47a1-a81c-96a35bfff901'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:02:00.602890 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-7479c04b-b1f2-47a1-a81c-96a35bfff901 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4da29c9977ca44baa867b3c7fc86164b', 'request_id': 'req-7479c04b-b1f2-47a1-a81c-96a35bfff901'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:02:00.608426 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-7479c04b-b1f2-47a1-a81c-96a35bfff901 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4da29c9977ca44baa867b3c7fc86164b', 'request_id': 'req-7479c04b-b1f2-47a1-a81c-96a35bfff901'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:02:00.620573 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-7479c04b-b1f2-47a1-a81c-96a35bfff901 admin admin] The token cache is being invalidate because project 4da29c9977ca44baa867b3c7fc86164b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81561) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:02:00.620859 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-7479c04b-b1f2-47a1-a81c-96a35bfff901 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:02:00.621484 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-7479c04b-b1f2-47a1-a81c-96a35bfff901 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:02:00.623009 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 932/1857] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:02:00 2022] DELETE /identity/v3/projects/4da29c9977ca44baa867b3c7fc86164b => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:02:00.628774 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-886316ae-34a8-4502-a307-f6ef3829d1b2 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:02:00.629089 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-886316ae-34a8-4502-a307-f6ef3829d1b2 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:02:00.629365 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-886316ae-34a8-4502-a307-f6ef3829d1b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:02:00.648700 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-886316ae-34a8-4502-a307-f6ef3829d1b2 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:02:00.649888 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-886316ae-34a8-4502-a307-f6ef3829d1b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:02:00.706841 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 926/1858] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:02:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:02:00.714571 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a49f5e6-72f0-4cbf-bc3d-7e92d5d7143a None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:02:00.714862 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a49f5e6-72f0-4cbf-bc3d-7e92d5d7143a None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:02:00.715405 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a49f5e6-72f0-4cbf-bc3d-7e92d5d7143a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:02:00.736221 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-4a49f5e6-72f0-4cbf-bc3d-7e92d5d7143a None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:02:00.737400 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-4a49f5e6-72f0-4cbf-bc3d-7e92d5d7143a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:02:00.745697 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 933/1859] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:02:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:02:00.749793 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-886316ae-34a8-4502-a307-f6ef3829d1b2 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:02:00.759329 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-886316ae-34a8-4502-a307-f6ef3829d1b2 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:02:00.763806 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4ed370-083e-411f-955c-87b8ffa08569 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:02:00.764396 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4ed370-083e-411f-955c-87b8ffa08569 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:02:00.764673 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4ed370-083e-411f-955c-87b8ffa08569 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:02:00.764942 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4ed370-083e-411f-955c-87b8ffa08569 admin admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:02:00.765582 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e4ed370-083e-411f-955c-87b8ffa08569 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 09:02:00.765884 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e4ed370-083e-411f-955c-87b8ffa08569 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:02:00.767510 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e4ed370-083e-411f-955c-87b8ffa08569 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:02:00.778943 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 927/1860] 10.222.0.36 () {58 vars in 1153 bytes} [Thu Oct 20 09:02:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 09:02:00.783050 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a49f5e6-72f0-4cbf-bc3d-7e92d5d7143a None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:02:00.792699 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a49f5e6-72f0-4cbf-bc3d-7e92d5d7143a None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:02:00.797198 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae57da2-5556-44dc-a359-f1f179e398ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:02:00.798320 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae57da2-5556-44dc-a359-f1f179e398ac admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:02:00.798320 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae57da2-5556-44dc-a359-f1f179e398ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:02:00.798320 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae57da2-5556-44dc-a359-f1f179e398ac admin admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:02:00.798960 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fae57da2-5556-44dc-a359-f1f179e398ac admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:02:00.800844 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fae57da2-5556-44dc-a359-f1f179e398ac admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:02:00.805771 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:02:00.828395 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 934/1861] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:02:00 2022] POST /identity/v3/projects => generated 375 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:02:00.832446 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4ed370-083e-411f-955c-87b8ffa08569 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:02:00.842652 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4ed370-083e-411f-955c-87b8ffa08569 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:02:00.890384 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:02:00.905219 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acbe107a-0bb8-4410-b8a7-dbc24a7d1770 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:02:00.906096 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acbe107a-0bb8-4410-b8a7-dbc24a7d1770 admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:02:00.906446 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acbe107a-0bb8-4410-b8a7-dbc24a7d1770 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:02:00.906814 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acbe107a-0bb8-4410-b8a7-dbc24a7d1770 admin admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:02:00.907527 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acbe107a-0bb8-4410-b8a7-dbc24a7d1770 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:02:00.909494 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acbe107a-0bb8-4410-b8a7-dbc24a7d1770 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:02:00.935236 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 928/1862] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:02:00 2022] POST /identity/v3/users => generated 288 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:02:00.940068 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae57da2-5556-44dc-a359-f1f179e398ac admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:02:00.955084 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae57da2-5556-44dc-a359-f1f179e398ac admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:02:00.960556 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aca631e-39e3-4883-bfc2-580f7e8aa185 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:02:00.961094 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aca631e-39e3-4883-bfc2-580f7e8aa185 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:02:00.961375 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aca631e-39e3-4883-bfc2-580f7e8aa185 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:02:00.961647 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aca631e-39e3-4883-bfc2-580f7e8aa185 admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:02:00.962299 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aca631e-39e3-4883-bfc2-580f7e8aa185 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:02:00.963930 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aca631e-39e3-4883-bfc2-580f7e8aa185 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:02:00.974663 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 935/1863] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:02:00 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:02:00.979284 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acbe107a-0bb8-4410-b8a7-dbc24a7d1770 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:02:00.989371 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acbe107a-0bb8-4410-b8a7-dbc24a7d1770 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:02:00.994189 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daf01bc6-f28f-49a5-9398-0ca6e144aefb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:02:00.994908 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf01bc6-f28f-49a5-9398-0ca6e144aefb admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:02:00.995253 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf01bc6-f28f-49a5-9398-0ca6e144aefb admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:02:00.995596 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf01bc6-f28f-49a5-9398-0ca6e144aefb admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:02:00.996332 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daf01bc6-f28f-49a5-9398-0ca6e144aefb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:02:00.998054 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daf01bc6-f28f-49a5-9398-0ca6e144aefb admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:02:01.011454 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 929/1864] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:02:00 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:02:01.015586 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aca631e-39e3-4883-bfc2-580f7e8aa185 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:02:01.027717 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aca631e-39e3-4883-bfc2-580f7e8aa185 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:02:01.034522 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78e7454b-ad57-4328-a1a9-27fe4500bf0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:02:01.036681 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e7454b-ad57-4328-a1a9-27fe4500bf0c admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:02:01.036681 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e7454b-ad57-4328-a1a9-27fe4500bf0c admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:02:01.036681 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e7454b-ad57-4328-a1a9-27fe4500bf0c admin admin] PATH_INFO: `/v3/projects/f1061324903c42ed9f0306471a113b82/users/be1f82d7a0a84fbebdd0dffef9f098ea/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:02:01.054515 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78e7454b-ad57-4328-a1a9-27fe4500bf0c admin admin] RBAC: Authorizing `identity:create_grant(project_id=f1061324903c42ed9f0306471a113b82, user_id=be1f82d7a0a84fbebdd0dffef9f098ea, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:02:01.057191 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78e7454b-ad57-4328-a1a9-27fe4500bf0c admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:02:01.070026 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 936/1865] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:02:01 2022] PUT /identity/v3/projects/f1061324903c42ed9f0306471a113b82/users/be1f82d7a0a84fbebdd0dffef9f098ea/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:02:01.075379 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a19362-2848-416e-afce-4056477ac37f None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:02:01.075668 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a19362-2848-416e-afce-4056477ac37f None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:02:01.075944 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a19362-2848-416e-afce-4056477ac37f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:02:01.095942 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-47a19362-2848-416e-afce-4056477ac37f None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:02:01.097150 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-47a19362-2848-416e-afce-4056477ac37f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:02:01.158121 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 930/1866] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:02:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 20 09:02:01.168465 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78e7454b-ad57-4328-a1a9-27fe4500bf0c admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:02:01.170193 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-78e7454b-ad57-4328-a1a9-27fe4500bf0c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:02:01.233509 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78e7454b-ad57-4328-a1a9-27fe4500bf0c admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:02:01.237304 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59b3a363-63c5-4d03-8f73-73af79667c7b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:02:01.237937 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b3a363-63c5-4d03-8f73-73af79667c7b service neutron] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:02:01.238218 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b3a363-63c5-4d03-8f73-73af79667c7b service neutron] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:02:01.238514 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b3a363-63c5-4d03-8f73-73af79667c7b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:02:01.248969 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59b3a363-63c5-4d03-8f73-73af79667c7b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:02:01.250419 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59b3a363-63c5-4d03-8f73-73af79667c7b service neutron] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:02:01.262491 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 937/1867] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:02:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:02:06.996045 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ae0761-a05f-4cc0-90f9-e5ecf3cd72a3 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:02:06.996532 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ae0761-a05f-4cc0-90f9-e5ecf3cd72a3 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:02:06.996627 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ae0761-a05f-4cc0-90f9-e5ecf3cd72a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:02:07.045048 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-b4ae0761-a05f-4cc0-90f9-e5ecf3cd72a3 None None] MFA Rules not processed for user `be1f82d7a0a84fbebdd0dffef9f098ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:02:07.047276 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-b4ae0761-a05f-4cc0-90f9-e5ecf3cd72a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:02:07.170959 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 931/1868] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:02:06 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:02:07.187683 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59b3a363-63c5-4d03-8f73-73af79667c7b service neutron] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:02:07.189632 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-59b3a363-63c5-4d03-8f73-73af79667c7b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:02:07.278069 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59b3a363-63c5-4d03-8f73-73af79667c7b service neutron] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:02:07.283599 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17bca245-2572-4c8d-906d-45f432653685 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a94e868115ec49918eda10724263930c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:02:07.284243 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17bca245-2572-4c8d-906d-45f432653685 service nova] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:02:07.284518 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17bca245-2572-4c8d-906d-45f432653685 service nova] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:02:07.284787 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17bca245-2572-4c8d-906d-45f432653685 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:02:07.295880 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17bca245-2572-4c8d-906d-45f432653685 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:02:07.297782 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17bca245-2572-4c8d-906d-45f432653685 service nova] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:02:07.309519 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 938/1869] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:02:07 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 122 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 20 09:02:08.262846 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd669d2-0eff-497f-bdac-ec604c56adb9 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:02:08.263432 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd669d2-0eff-497f-bdac-ec604c56adb9 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:02:08.263432 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd669d2-0eff-497f-bdac-ec604c56adb9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:02:08.282590 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-3bd669d2-0eff-497f-bdac-ec604c56adb9 None None] MFA Rules not processed for user `930afcdb977f4ed2a33b366563a755f1`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:02:08.283819 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-3bd669d2-0eff-497f-bdac-ec604c56adb9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:02:08.354725 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 932/1870] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:02:08 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:02:09.337733 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b98a848a-d9a0-400c-9f08-377085a757a0 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:02:09.338589 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b98a848a-d9a0-400c-9f08-377085a757a0 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:02:09.339079 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b98a848a-d9a0-400c-9f08-377085a757a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:02:09.387669 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-b98a848a-d9a0-400c-9f08-377085a757a0 None None] MFA Rules not processed for user `be1f82d7a0a84fbebdd0dffef9f098ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:02:09.388927 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-b98a848a-d9a0-400c-9f08-377085a757a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:02:09.415697 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 939/1871] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:02:09 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 76 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:02:18.344547 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=96,keystone:INSERT=3 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:02:19.313640 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bd669d2-0eff-497f-bdac-ec604c56adb9 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:02:19.315082 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-3bd669d2-0eff-497f-bdac-ec604c56adb9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:02:19.330584 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bd669d2-0eff-497f-bdac-ec604c56adb9 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:02:19.335828 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c0ef35-c374-43d0-9256-ef2056f03884 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:02:19.336534 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c0ef35-c374-43d0-9256-ef2056f03884 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:02:19.336881 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c0ef35-c374-43d0-9256-ef2056f03884 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:02:19.337242 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c0ef35-c374-43d0-9256-ef2056f03884 admin admin] PATH_INFO: `/v3/users/a6e64de6ce4c484fb461ec4d0d8a60a3` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:02:19.338431 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02c0ef35-c374-43d0-9256-ef2056f03884 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a6e64de6ce4c484fb461ec4d0d8a60a3)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:02:19.341025 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02c0ef35-c374-43d0-9256-ef2056f03884 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:02:19.368238 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=84,keystone:INSERT=2 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:02:19.407916 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-02c0ef35-c374-43d0-9256-ef2056f03884 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a6e64de6ce4c484fb461ec4d0d8a60a3'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:02:19.421322 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-02c0ef35-c374-43d0-9256-ef2056f03884 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a6e64de6ce4c484fb461ec4d0d8a60a3'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:02:19.439681 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-02c0ef35-c374-43d0-9256-ef2056f03884 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a6e64de6ce4c484fb461ec4d0d8a60a3'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:02:19.458109 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-02c0ef35-c374-43d0-9256-ef2056f03884 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a6e64de6ce4c484fb461ec4d0d8a60a3'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:02:19.459500 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-02c0ef35-c374-43d0-9256-ef2056f03884 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a6e64de6ce4c484fb461ec4d0d8a60a3'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:02:19.468867 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 933/1872] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:02:19 2022] DELETE /identity/v3/users/a6e64de6ce4c484fb461ec4d0d8a60a3 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:02:19.473776 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7123a8e-dbb0-4123-86e7-a168eac2b730 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:02:19.474782 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7123a8e-dbb0-4123-86e7-a168eac2b730 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:02:19.475265 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7123a8e-dbb0-4123-86e7-a168eac2b730 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:02:19.501640 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-c7123a8e-dbb0-4123-86e7-a168eac2b730 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:02:19.503293 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-c7123a8e-dbb0-4123-86e7-a168eac2b730 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:02:19.602101 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 940/1873] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:02:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:02:19.621223 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c0ef35-c374-43d0-9256-ef2056f03884 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:02:19.623312 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-02c0ef35-c374-43d0-9256-ef2056f03884 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:02:19.695600 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c0ef35-c374-43d0-9256-ef2056f03884 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:02:19.700295 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98700406-ec00-48fc-8adf-9661ec337a74 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:02:19.700953 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98700406-ec00-48fc-8adf-9661ec337a74 service neutron] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:02:19.701235 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98700406-ec00-48fc-8adf-9661ec337a74 service neutron] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:02:19.701635 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98700406-ec00-48fc-8adf-9661ec337a74 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:02:19.714719 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98700406-ec00-48fc-8adf-9661ec337a74 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:02:19.716610 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98700406-ec00-48fc-8adf-9661ec337a74 service neutron] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:02:19.730025 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 934/1874] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:02:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:02:20.166980 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7123a8e-dbb0-4123-86e7-a168eac2b730 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:02:20.168861 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-c7123a8e-dbb0-4123-86e7-a168eac2b730 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:02:20.185861 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7123a8e-dbb0-4123-86e7-a168eac2b730 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:02:20.190853 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7be231e7-37f9-434c-85d3-de609eee7833 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:02:20.191543 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be231e7-37f9-434c-85d3-de609eee7833 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:02:20.191827 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be231e7-37f9-434c-85d3-de609eee7833 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:02:20.192099 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be231e7-37f9-434c-85d3-de609eee7833 admin admin] PATH_INFO: `/v3/projects/f5bb1132fe1e4780946808c8b7c59479` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:02:20.193223 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7be231e7-37f9-434c-85d3-de609eee7833 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f5bb1132fe1e4780946808c8b7c59479)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:02:20.195344 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7be231e7-37f9-434c-85d3-de609eee7833 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:02:20.243323 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-7be231e7-37f9-434c-85d3-de609eee7833 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f5bb1132fe1e4780946808c8b7c59479', 'request_id': 'req-7be231e7-37f9-434c-85d3-de609eee7833'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:02:20.249742 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-7be231e7-37f9-434c-85d3-de609eee7833 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f5bb1132fe1e4780946808c8b7c59479', 'request_id': 'req-7be231e7-37f9-434c-85d3-de609eee7833'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:02:20.255977 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-7be231e7-37f9-434c-85d3-de609eee7833 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f5bb1132fe1e4780946808c8b7c59479', 'request_id': 'req-7be231e7-37f9-434c-85d3-de609eee7833'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:02:20.268799 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-7be231e7-37f9-434c-85d3-de609eee7833 admin admin] The token cache is being invalidate because project f5bb1132fe1e4780946808c8b7c59479 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81561) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:02:20.269115 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-7be231e7-37f9-434c-85d3-de609eee7833 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:02:20.269722 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-7be231e7-37f9-434c-85d3-de609eee7833 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:02:20.271259 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 941/1875] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:02:20 2022] DELETE /identity/v3/projects/f5bb1132fe1e4780946808c8b7c59479 => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:02:29.724466 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=55,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:02:30.265597 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=54,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:02:42.171543 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f7677a-4d36-4230-94e6-4005c921e63c None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:02:42.172038 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f7677a-4d36-4230-94e6-4005c921e63c None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:02:42.172038 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f7677a-4d36-4230-94e6-4005c921e63c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:02:42.194341 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-06f7677a-4d36-4230-94e6-4005c921e63c None None] MFA Rules not processed for user `f4b0e9187ce74f7888e9e7094be5c824`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:02:42.195774 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-06f7677a-4d36-4230-94e6-4005c921e63c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:02:42.264147 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 935/1876] 10.222.0.36 () {58 vars in 1009 bytes} [Thu Oct 20 09:02:42 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:02:42.308779 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7be231e7-37f9-434c-85d3-de609eee7833 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:02:42.310249 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-7be231e7-37f9-434c-85d3-de609eee7833 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:02:42.400735 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7be231e7-37f9-434c-85d3-de609eee7833 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:02:42.406202 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f876c96d-0bb8-45c9-a0ef-efa2772b7657 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4d558918cd8b4ebea4fe8765297304db', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:02:42.406854 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f876c96d-0bb8-45c9-a0ef-efa2772b7657 service swift] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:02:42.407132 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f876c96d-0bb8-45c9-a0ef-efa2772b7657 service swift] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:02:42.407399 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f876c96d-0bb8-45c9-a0ef-efa2772b7657 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:02:42.416284 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f876c96d-0bb8-45c9-a0ef-efa2772b7657 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:02:42.417904 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f876c96d-0bb8-45c9-a0ef-efa2772b7657 service swift] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:02:42.426123 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 942/1877] 10.222.0.36 () {60 vars in 1384 bytes} [Thu Oct 20 09:02:42 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 20 09:02:44.064123 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad074353-8650-4d90-9f85-9cc18e60109b None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:02:44.064635 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad074353-8650-4d90-9f85-9cc18e60109b None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:02:44.064871 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad074353-8650-4d90-9f85-9cc18e60109b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:02:44.090392 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-ad074353-8650-4d90-9f85-9cc18e60109b None None] MFA Rules not processed for user `8002ab26f6d64822aaa65aa730781b76`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:02:44.092248 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-ad074353-8650-4d90-9f85-9cc18e60109b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:02:44.224289 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 936/1878] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:02:44 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:02:52.422797 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=32 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:02:54.191143 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=50 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:02:54.890154 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7520697d-c24d-48d2-bba3-b0752fc04431 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:02:54.890756 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7520697d-c24d-48d2-bba3-b0752fc04431 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:02:54.891191 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7520697d-c24d-48d2-bba3-b0752fc04431 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:02:54.915151 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-7520697d-c24d-48d2-bba3-b0752fc04431 None None] MFA Rules not processed for user `7f932695a5994c5790d14c210ea65326`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:02:54.916937 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-7520697d-c24d-48d2-bba3-b0752fc04431 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:02:54.999725 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 943/1879] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:02:54 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:03:02.320861 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6309daca-c291-4edb-8cc6-80ca9dbacee5 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:03:02.321351 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6309daca-c291-4edb-8cc6-80ca9dbacee5 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:03:02.321564 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6309daca-c291-4edb-8cc6-80ca9dbacee5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:03:02.341589 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-6309daca-c291-4edb-8cc6-80ca9dbacee5 None None] MFA Rules not processed for user `f4b0e9187ce74f7888e9e7094be5c824`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:03:02.342762 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-6309daca-c291-4edb-8cc6-80ca9dbacee5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:03:02.349818 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 937/1880] 10.222.0.36 () {58 vars in 967 bytes} [Thu Oct 20 09:03:02 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:03:02.369491 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7520697d-c24d-48d2-bba3-b0752fc04431 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:03:02.389685 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7520697d-c24d-48d2-bba3-b0752fc04431 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:03:02.394997 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1d84b2b-090b-472c-b65c-79161b834a1e service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4d558918cd8b4ebea4fe8765297304db', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:03:02.395610 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d84b2b-090b-472c-b65c-79161b834a1e service swift] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:03:02.395995 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d84b2b-090b-472c-b65c-79161b834a1e service swift] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:03:02.396268 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d84b2b-090b-472c-b65c-79161b834a1e service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:03:02.404842 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1d84b2b-090b-472c-b65c-79161b834a1e service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:03:02.406490 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1d84b2b-090b-472c-b65c-79161b834a1e service swift] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:03:02.415403 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 944/1881] 10.222.0.36 () {60 vars in 1384 bytes} [Thu Oct 20 09:03:02 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 20 09:03:06.815421 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-321df3e8-c113-4143-9216-7a3aef77d4fd None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:03:06.815910 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-321df3e8-c113-4143-9216-7a3aef77d4fd None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:03:06.816337 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-321df3e8-c113-4143-9216-7a3aef77d4fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:03:06.841213 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-321df3e8-c113-4143-9216-7a3aef77d4fd None None] MFA Rules not processed for user `f4b0e9187ce74f7888e9e7094be5c824`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:03:06.842584 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-321df3e8-c113-4143-9216-7a3aef77d4fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:03:06.850109 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 938/1882] 10.222.0.36 () {58 vars in 1009 bytes} [Thu Oct 20 09:03:06 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 20 09:03:06.863389 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1d84b2b-090b-472c-b65c-79161b834a1e service swift] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:03:06.876359 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1d84b2b-090b-472c-b65c-79161b834a1e service swift] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:03:06.881239 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4885c76-f36d-4f54-876b-6faed8fa6325 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4d558918cd8b4ebea4fe8765297304db', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:03:06.881833 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4885c76-f36d-4f54-876b-6faed8fa6325 service swift] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:03:06.882137 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4885c76-f36d-4f54-876b-6faed8fa6325 service swift] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:03:06.882452 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4885c76-f36d-4f54-876b-6faed8fa6325 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:03:06.890584 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4885c76-f36d-4f54-876b-6faed8fa6325 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:03:06.892146 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4885c76-f36d-4f54-876b-6faed8fa6325 service swift] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:03:06.900526 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 945/1883] 10.222.0.36 () {60 vars in 1384 bytes} [Thu Oct 20 09:03:06 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 20 09:03:07.887432 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-321df3e8-c113-4143-9216-7a3aef77d4fd None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:03:07.888950 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-321df3e8-c113-4143-9216-7a3aef77d4fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:03:07.980384 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-321df3e8-c113-4143-9216-7a3aef77d4fd None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:03:07.984936 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c05631d-a9e6-4186-af67-5fe942ee85e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:03:07.985573 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c05631d-a9e6-4186-af67-5fe942ee85e1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:03:07.985853 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c05631d-a9e6-4186-af67-5fe942ee85e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:03:07.986130 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c05631d-a9e6-4186-af67-5fe942ee85e1 admin admin] PATH_INFO: `/v3/users/930afcdb977f4ed2a33b366563a755f1` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:03:07.987347 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c05631d-a9e6-4186-af67-5fe942ee85e1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=930afcdb977f4ed2a33b366563a755f1)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:03:07.989306 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c05631d-a9e6-4186-af67-5fe942ee85e1 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:03:08.050637 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-2c05631d-a9e6-4186-af67-5fe942ee85e1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '930afcdb977f4ed2a33b366563a755f1'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:03:08.060016 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-2c05631d-a9e6-4186-af67-5fe942ee85e1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '930afcdb977f4ed2a33b366563a755f1'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:03:08.073641 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-2c05631d-a9e6-4186-af67-5fe942ee85e1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '930afcdb977f4ed2a33b366563a755f1'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:03:08.081812 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-2c05631d-a9e6-4186-af67-5fe942ee85e1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '930afcdb977f4ed2a33b366563a755f1'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:03:08.082573 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-2c05631d-a9e6-4186-af67-5fe942ee85e1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '930afcdb977f4ed2a33b366563a755f1'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:03:08.089342 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 939/1884] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:03:07 2022] DELETE /identity/v3/users/930afcdb977f4ed2a33b366563a755f1 => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:03:08.341503 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4885c76-f36d-4f54-876b-6faed8fa6325 service swift] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:03:08.359229 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4885c76-f36d-4f54-876b-6faed8fa6325 service swift] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:03:08.497364 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1075c7b6-295d-49f5-8946-8d7b797f7b3f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:03:08.498260 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1075c7b6-295d-49f5-8946-8d7b797f7b3f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:03:08.498741 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1075c7b6-295d-49f5-8946-8d7b797f7b3f admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:03:08.499187 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1075c7b6-295d-49f5-8946-8d7b797f7b3f admin admin] PATH_INFO: `/v3/projects/847f0b8423334fc7b505313a10845bcb` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:03:08.500359 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1075c7b6-295d-49f5-8946-8d7b797f7b3f admin admin] RBAC: Authorizing `identity:delete_project(project_id=847f0b8423334fc7b505313a10845bcb)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:03:08.502277 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1075c7b6-295d-49f5-8946-8d7b797f7b3f admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:03:08.592300 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-1075c7b6-295d-49f5-8946-8d7b797f7b3f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '847f0b8423334fc7b505313a10845bcb', 'request_id': 'req-1075c7b6-295d-49f5-8946-8d7b797f7b3f'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:03:08.601517 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-1075c7b6-295d-49f5-8946-8d7b797f7b3f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '847f0b8423334fc7b505313a10845bcb', 'request_id': 'req-1075c7b6-295d-49f5-8946-8d7b797f7b3f'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:03:08.610016 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-1075c7b6-295d-49f5-8946-8d7b797f7b3f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '847f0b8423334fc7b505313a10845bcb', 'request_id': 'req-1075c7b6-295d-49f5-8946-8d7b797f7b3f'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:03:08.636748 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-1075c7b6-295d-49f5-8946-8d7b797f7b3f admin admin] The token cache is being invalidate because project 847f0b8423334fc7b505313a10845bcb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81561) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:03:08.638273 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-1075c7b6-295d-49f5-8946-8d7b797f7b3f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:03:08.639292 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-1075c7b6-295d-49f5-8946-8d7b797f7b3f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:03:08.648082 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 946/1885] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:03:08 2022] DELETE /identity/v3/projects/847f0b8423334fc7b505313a10845bcb => generated 0 bytes in 307 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:03:08.653867 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ec108c-7049-449e-bc21-34fd7b08bbe5 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:03:08.654576 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ec108c-7049-449e-bc21-34fd7b08bbe5 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:03:08.654960 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ec108c-7049-449e-bc21-34fd7b08bbe5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:03:08.692536 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-13ec108c-7049-449e-bc21-34fd7b08bbe5 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:03:08.694011 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-13ec108c-7049-449e-bc21-34fd7b08bbe5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:03:08.857064 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 940/1886] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:03:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 204 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:03:08.865143 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b06c1242-33b7-43b1-ade9-debf4e474888 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:03:08.865429 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b06c1242-33b7-43b1-ade9-debf4e474888 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:03:08.865708 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b06c1242-33b7-43b1-ade9-debf4e474888 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:03:08.921315 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-b06c1242-33b7-43b1-ade9-debf4e474888 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:03:08.922579 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-b06c1242-33b7-43b1-ade9-debf4e474888 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:03:08.931713 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 947/1887] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:03:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 68 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 20 09:03:08.936603 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13ec108c-7049-449e-bc21-34fd7b08bbe5 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:03:08.954036 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13ec108c-7049-449e-bc21-34fd7b08bbe5 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:03:08.959912 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab89217b-b686-4cbc-bb47-ec37e4a2b208 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:03:08.960657 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab89217b-b686-4cbc-bb47-ec37e4a2b208 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:03:08.960958 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab89217b-b686-4cbc-bb47-ec37e4a2b208 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:03:08.961216 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab89217b-b686-4cbc-bb47-ec37e4a2b208 admin admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:03:08.961880 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab89217b-b686-4cbc-bb47-ec37e4a2b208 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 09:03:08.962182 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab89217b-b686-4cbc-bb47-ec37e4a2b208 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:03:08.964104 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab89217b-b686-4cbc-bb47-ec37e4a2b208 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:03:08.966328 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:03:08.980290 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 941/1888] 10.222.0.36 () {58 vars in 1153 bytes} [Thu Oct 20 09:03:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 09:03:08.985010 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b06c1242-33b7-43b1-ade9-debf4e474888 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:03:08.989794 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:03:09.000338 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b06c1242-33b7-43b1-ade9-debf4e474888 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:03:09.014501 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9196c38b-ea69-4645-854d-ec1607e4cec4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:03:09.015786 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9196c38b-ea69-4645-854d-ec1607e4cec4 admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:03:09.016343 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9196c38b-ea69-4645-854d-ec1607e4cec4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:03:09.016827 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9196c38b-ea69-4645-854d-ec1607e4cec4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:03:09.018188 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9196c38b-ea69-4645-854d-ec1607e4cec4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:03:09.020697 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9196c38b-ea69-4645-854d-ec1607e4cec4 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:03:09.065567 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 948/1889] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:03:08 2022] POST /identity/v3/projects => generated 373 bytes in 81 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:03:09.070800 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab89217b-b686-4cbc-bb47-ec37e4a2b208 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:03:09.087377 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab89217b-b686-4cbc-bb47-ec37e4a2b208 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:03:09.205984 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff1830ec-ef95-4a01-a462-b7ecef50d1d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:03:09.206804 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff1830ec-ef95-4a01-a462-b7ecef50d1d1 admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:03:09.207189 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff1830ec-ef95-4a01-a462-b7ecef50d1d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:03:09.207571 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff1830ec-ef95-4a01-a462-b7ecef50d1d1 admin admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:03:09.208422 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff1830ec-ef95-4a01-a462-b7ecef50d1d1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:03:09.210532 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff1830ec-ef95-4a01-a462-b7ecef50d1d1 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:03:09.238262 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 942/1890] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:03:09 2022] POST /identity/v3/users => generated 287 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 20 09:03:09.243193 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9196c38b-ea69-4645-854d-ec1607e4cec4 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:03:09.254281 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9196c38b-ea69-4645-854d-ec1607e4cec4 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:03:09.261202 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2925af8b-170d-476b-9fcf-7c3a5e644e01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:03:09.262106 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2925af8b-170d-476b-9fcf-7c3a5e644e01 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:03:09.262463 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2925af8b-170d-476b-9fcf-7c3a5e644e01 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:03:09.262835 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2925af8b-170d-476b-9fcf-7c3a5e644e01 admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:03:09.263574 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2925af8b-170d-476b-9fcf-7c3a5e644e01 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:03:09.265303 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2925af8b-170d-476b-9fcf-7c3a5e644e01 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:03:09.278011 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 949/1891] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:03:09 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 20 09:03:09.281806 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff1830ec-ef95-4a01-a462-b7ecef50d1d1 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:03:09.292882 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff1830ec-ef95-4a01-a462-b7ecef50d1d1 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:03:09.298753 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fa9906d-c345-4c2c-bb86-5d1ed6dea945 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:03:09.299428 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa9906d-c345-4c2c-bb86-5d1ed6dea945 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:03:09.299779 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa9906d-c345-4c2c-bb86-5d1ed6dea945 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:03:09.300058 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa9906d-c345-4c2c-bb86-5d1ed6dea945 admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:03:09.300741 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fa9906d-c345-4c2c-bb86-5d1ed6dea945 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:03:09.302517 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fa9906d-c345-4c2c-bb86-5d1ed6dea945 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:03:09.317852 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 943/1892] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:03:09 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:03:09.319291 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2925af8b-170d-476b-9fcf-7c3a5e644e01 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:03:09.335319 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2925af8b-170d-476b-9fcf-7c3a5e644e01 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:03:09.340913 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e7b6f7b-0c3d-4986-bf52-16dc72ca2c6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:03:09.341794 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7b6f7b-0c3d-4986-bf52-16dc72ca2c6e admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:03:09.342810 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7b6f7b-0c3d-4986-bf52-16dc72ca2c6e admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:03:09.343264 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7b6f7b-0c3d-4986-bf52-16dc72ca2c6e admin admin] PATH_INFO: `/v3/projects/bf8bf4571f4b4090bff58f73fe1ee3ff/users/46441a793daa481090fac3f63a97d7cd/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:03:09.360426 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e7b6f7b-0c3d-4986-bf52-16dc72ca2c6e admin admin] RBAC: Authorizing `identity:create_grant(project_id=bf8bf4571f4b4090bff58f73fe1ee3ff, user_id=46441a793daa481090fac3f63a97d7cd, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:03:09.363327 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e7b6f7b-0c3d-4986-bf52-16dc72ca2c6e admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:03:09.384138 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 950/1893] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:03:09 2022] PUT /identity/v3/projects/bf8bf4571f4b4090bff58f73fe1ee3ff/users/46441a793daa481090fac3f63a97d7cd/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:03:09.395332 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0592c408-b774-4125-ac83-878b159c25f2 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:03:09.395853 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0592c408-b774-4125-ac83-878b159c25f2 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:03:09.396130 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0592c408-b774-4125-ac83-878b159c25f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:03:09.430624 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-0592c408-b774-4125-ac83-878b159c25f2 None None] MFA Rules not processed for user `46441a793daa481090fac3f63a97d7cd`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:03:09.431803 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-0592c408-b774-4125-ac83-878b159c25f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:03:09.515411 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 944/1894] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:03:09 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:03:09.526413 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e7b6f7b-0c3d-4986-bf52-16dc72ca2c6e admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:03:09.530087 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-4e7b6f7b-0c3d-4986-bf52-16dc72ca2c6e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:03:09.599494 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e7b6f7b-0c3d-4986-bf52-16dc72ca2c6e admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:03:09.603439 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7977c00e-18b0-4e5d-b2ca-49f37c7bdc1c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:03:09.604054 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7977c00e-18b0-4e5d-b2ca-49f37c7bdc1c service cinder] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:03:09.604368 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7977c00e-18b0-4e5d-b2ca-49f37c7bdc1c service cinder] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:03:09.607879 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7977c00e-18b0-4e5d-b2ca-49f37c7bdc1c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:03:09.617939 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7977c00e-18b0-4e5d-b2ca-49f37c7bdc1c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:03:09.619520 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7977c00e-18b0-4e5d-b2ca-49f37c7bdc1c service cinder] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:03:09.635074 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 951/1895] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:03:09 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:03:19.511204 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=69,keystone:INSERT=3 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:03:19.629083 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=49,keystone:INSERT=2 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:03:21.068262 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0592c408-b774-4125-ac83-878b159c25f2 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:03:21.129082 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0592c408-b774-4125-ac83-878b159c25f2 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:03:21.134158 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e0151bb-9b20-4d2d-b897-b1dcefbc5861 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:03:21.134829 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0151bb-9b20-4d2d-b897-b1dcefbc5861 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:03:21.135133 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0151bb-9b20-4d2d-b897-b1dcefbc5861 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:03:21.135407 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0151bb-9b20-4d2d-b897-b1dcefbc5861 admin admin] PATH_INFO: `/v3/users/46441a793daa481090fac3f63a97d7cd` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:03:21.136578 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e0151bb-9b20-4d2d-b897-b1dcefbc5861 admin admin] RBAC: Authorizing `identity:delete_user(user_id=46441a793daa481090fac3f63a97d7cd)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:03:21.138566 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e0151bb-9b20-4d2d-b897-b1dcefbc5861 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:03:21.185412 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-1e0151bb-9b20-4d2d-b897-b1dcefbc5861 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '46441a793daa481090fac3f63a97d7cd'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:03:21.192396 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-1e0151bb-9b20-4d2d-b897-b1dcefbc5861 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '46441a793daa481090fac3f63a97d7cd'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:03:21.205029 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-1e0151bb-9b20-4d2d-b897-b1dcefbc5861 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '46441a793daa481090fac3f63a97d7cd'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:03:21.212410 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-1e0151bb-9b20-4d2d-b897-b1dcefbc5861 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '46441a793daa481090fac3f63a97d7cd'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:03:21.213055 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-1e0151bb-9b20-4d2d-b897-b1dcefbc5861 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '46441a793daa481090fac3f63a97d7cd'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:03:21.219342 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 945/1896] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:03:21 2022] DELETE /identity/v3/users/46441a793daa481090fac3f63a97d7cd => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:03:21.223655 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29321c75-08b0-43ec-8895-05c5dd0ff317 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:03:21.224123 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29321c75-08b0-43ec-8895-05c5dd0ff317 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:03:21.224231 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29321c75-08b0-43ec-8895-05c5dd0ff317 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:03:21.244217 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-29321c75-08b0-43ec-8895-05c5dd0ff317 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:03:21.245733 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-29321c75-08b0-43ec-8895-05c5dd0ff317 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:03:21.304357 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 952/1897] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:03:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:03:21.313870 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e0151bb-9b20-4d2d-b897-b1dcefbc5861 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:03:21.315317 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-1e0151bb-9b20-4d2d-b897-b1dcefbc5861 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:03:21.372113 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e0151bb-9b20-4d2d-b897-b1dcefbc5861 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:03:21.375838 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d72b2782-725f-4282-b6e5-97b9c68e568d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:03:21.376449 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d72b2782-725f-4282-b6e5-97b9c68e568d service neutron] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:03:21.376730 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d72b2782-725f-4282-b6e5-97b9c68e568d service neutron] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:03:21.377003 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d72b2782-725f-4282-b6e5-97b9c68e568d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:03:21.386198 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d72b2782-725f-4282-b6e5-97b9c68e568d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:03:21.387662 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d72b2782-725f-4282-b6e5-97b9c68e568d service neutron] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:03:21.397156 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 946/1898] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:03:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:03:21.668545 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29321c75-08b0-43ec-8895-05c5dd0ff317 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:03:21.670044 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-29321c75-08b0-43ec-8895-05c5dd0ff317 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:03:21.684703 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29321c75-08b0-43ec-8895-05c5dd0ff317 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:03:21.690310 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ee97df5-11fb-45ef-b453-a7a67adeaf3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:03:21.690998 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee97df5-11fb-45ef-b453-a7a67adeaf3e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:03:21.691255 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee97df5-11fb-45ef-b453-a7a67adeaf3e admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:03:21.691650 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee97df5-11fb-45ef-b453-a7a67adeaf3e admin admin] PATH_INFO: `/v3/projects/bf8bf4571f4b4090bff58f73fe1ee3ff` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:03:21.692740 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ee97df5-11fb-45ef-b453-a7a67adeaf3e admin admin] RBAC: Authorizing `identity:delete_project(project_id=bf8bf4571f4b4090bff58f73fe1ee3ff)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:03:21.695128 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ee97df5-11fb-45ef-b453-a7a67adeaf3e admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:03:21.741481 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-0ee97df5-11fb-45ef-b453-a7a67adeaf3e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bf8bf4571f4b4090bff58f73fe1ee3ff', 'request_id': 'req-0ee97df5-11fb-45ef-b453-a7a67adeaf3e'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:03:21.748265 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-0ee97df5-11fb-45ef-b453-a7a67adeaf3e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bf8bf4571f4b4090bff58f73fe1ee3ff', 'request_id': 'req-0ee97df5-11fb-45ef-b453-a7a67adeaf3e'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:03:21.753408 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-0ee97df5-11fb-45ef-b453-a7a67adeaf3e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bf8bf4571f4b4090bff58f73fe1ee3ff', 'request_id': 'req-0ee97df5-11fb-45ef-b453-a7a67adeaf3e'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:03:21.767168 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-0ee97df5-11fb-45ef-b453-a7a67adeaf3e admin admin] The token cache is being invalidate because project bf8bf4571f4b4090bff58f73fe1ee3ff was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81561) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:03:21.767457 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-0ee97df5-11fb-45ef-b453-a7a67adeaf3e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:03:21.768123 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-0ee97df5-11fb-45ef-b453-a7a67adeaf3e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:03:21.769681 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 953/1899] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:03:21 2022] DELETE /identity/v3/projects/bf8bf4571f4b4090bff58f73fe1ee3ff => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:03:21.775454 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9874fbe2-ea24-4d5c-9d30-1929e3dd42e5 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:03:21.775936 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9874fbe2-ea24-4d5c-9d30-1929e3dd42e5 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:03:21.776281 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9874fbe2-ea24-4d5c-9d30-1929e3dd42e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:03:21.796256 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-9874fbe2-ea24-4d5c-9d30-1929e3dd42e5 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:03:21.797461 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-9874fbe2-ea24-4d5c-9d30-1929e3dd42e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:03:21.814313 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:03:21.860589 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 947/1900] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:03:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:03:21.869112 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87fd4359-5df6-4cbc-b987-b843a2ac3c5a None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:03:21.869400 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87fd4359-5df6-4cbc-b987-b843a2ac3c5a None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:03:21.869675 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87fd4359-5df6-4cbc-b987-b843a2ac3c5a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:03:21.889580 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-87fd4359-5df6-4cbc-b987-b843a2ac3c5a None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:03:21.890821 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-87fd4359-5df6-4cbc-b987-b843a2ac3c5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:03:21.899259 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 954/1901] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:03:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:03:21.903463 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9874fbe2-ea24-4d5c-9d30-1929e3dd42e5 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:03:21.912970 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9874fbe2-ea24-4d5c-9d30-1929e3dd42e5 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:03:21.917473 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eea6fa9-fdac-4df1-8bed-c0cd06c2a6c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:03:21.918066 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eea6fa9-fdac-4df1-8bed-c0cd06c2a6c8 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:03:21.918454 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eea6fa9-fdac-4df1-8bed-c0cd06c2a6c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:03:21.918755 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eea6fa9-fdac-4df1-8bed-c0cd06c2a6c8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:03:21.919391 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eea6fa9-fdac-4df1-8bed-c0cd06c2a6c8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 09:03:21.919691 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eea6fa9-fdac-4df1-8bed-c0cd06c2a6c8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:03:21.921283 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eea6fa9-fdac-4df1-8bed-c0cd06c2a6c8 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:03:21.933267 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 948/1902] 10.222.0.36 () {58 vars in 1153 bytes} [Thu Oct 20 09:03:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 09:03:21.937386 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87fd4359-5df6-4cbc-b987-b843a2ac3c5a None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:03:21.947201 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87fd4359-5df6-4cbc-b987-b843a2ac3c5a None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:03:21.951740 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a70dd93-fa82-4c9f-9008-74a103fecef9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:03:21.952323 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a70dd93-fa82-4c9f-9008-74a103fecef9 admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:03:21.952594 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a70dd93-fa82-4c9f-9008-74a103fecef9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:03:21.952857 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a70dd93-fa82-4c9f-9008-74a103fecef9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:03:21.953486 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a70dd93-fa82-4c9f-9008-74a103fecef9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:03:21.955405 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a70dd93-fa82-4c9f-9008-74a103fecef9 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:03:21.981539 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 955/1903] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:03:21 2022] POST /identity/v3/projects => generated 379 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 20 09:03:21.985736 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eea6fa9-fdac-4df1-8bed-c0cd06c2a6c8 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:03:21.996050 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eea6fa9-fdac-4df1-8bed-c0cd06c2a6c8 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:03:22.049774 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7e6a7cc-1fe5-454a-9851-1a00baab2da9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:03:22.050294 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e6a7cc-1fe5-454a-9851-1a00baab2da9 admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:03:22.050600 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e6a7cc-1fe5-454a-9851-1a00baab2da9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:03:22.050871 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e6a7cc-1fe5-454a-9851-1a00baab2da9 admin admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:03:22.051494 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7e6a7cc-1fe5-454a-9851-1a00baab2da9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:03:22.053920 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7e6a7cc-1fe5-454a-9851-1a00baab2da9 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:03:22.075112 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 949/1904] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:03:21 2022] POST /identity/v3/users => generated 290 bytes in 90 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:03:22.079232 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a70dd93-fa82-4c9f-9008-74a103fecef9 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:03:22.089296 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a70dd93-fa82-4c9f-9008-74a103fecef9 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:03:22.094065 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc9fe07f-cfab-4144-8b98-c7de167a1b33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:03:22.094679 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc9fe07f-cfab-4144-8b98-c7de167a1b33 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:03:22.094955 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc9fe07f-cfab-4144-8b98-c7de167a1b33 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:03:22.095218 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc9fe07f-cfab-4144-8b98-c7de167a1b33 admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:03:22.095843 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc9fe07f-cfab-4144-8b98-c7de167a1b33 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:03:22.097436 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc9fe07f-cfab-4144-8b98-c7de167a1b33 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:03:22.107111 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 956/1905] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:03:22 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:03:22.110727 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7e6a7cc-1fe5-454a-9851-1a00baab2da9 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:03:22.120945 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7e6a7cc-1fe5-454a-9851-1a00baab2da9 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:03:22.125506 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c9e9b44-3ebb-44e4-a603-a4e6179b54f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:03:22.126078 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c9e9b44-3ebb-44e4-a603-a4e6179b54f0 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:03:22.126355 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c9e9b44-3ebb-44e4-a603-a4e6179b54f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:03:22.126658 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c9e9b44-3ebb-44e4-a603-a4e6179b54f0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:03:22.127287 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c9e9b44-3ebb-44e4-a603-a4e6179b54f0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:03:22.128868 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c9e9b44-3ebb-44e4-a603-a4e6179b54f0 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:03:22.140201 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 950/1906] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:03:22 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 20 09:03:22.144219 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc9fe07f-cfab-4144-8b98-c7de167a1b33 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:03:22.154392 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc9fe07f-cfab-4144-8b98-c7de167a1b33 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:03:22.159379 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35287ec2-4f9c-4a96-bd53-aab2b6b644e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:03:22.160021 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35287ec2-4f9c-4a96-bd53-aab2b6b644e1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:03:22.160295 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35287ec2-4f9c-4a96-bd53-aab2b6b644e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:03:22.160562 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35287ec2-4f9c-4a96-bd53-aab2b6b644e1 admin admin] PATH_INFO: `/v3/projects/98827dbef155480b9f26792b70014b4b/users/65ca2eae6db24bb094afaacdfb5c1980/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:03:22.174976 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35287ec2-4f9c-4a96-bd53-aab2b6b644e1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=98827dbef155480b9f26792b70014b4b, user_id=65ca2eae6db24bb094afaacdfb5c1980, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:03:22.177613 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35287ec2-4f9c-4a96-bd53-aab2b6b644e1 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:03:22.191008 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 957/1907] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:03:22 2022] PUT /identity/v3/projects/98827dbef155480b9f26792b70014b4b/users/65ca2eae6db24bb094afaacdfb5c1980/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:03:22.196594 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75b9f186-f9b3-41e9-845f-861eb12cde93 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:03:22.196883 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75b9f186-f9b3-41e9-845f-861eb12cde93 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:03:22.197182 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75b9f186-f9b3-41e9-845f-861eb12cde93 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:03:22.217946 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-75b9f186-f9b3-41e9-845f-861eb12cde93 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:03:22.219454 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-75b9f186-f9b3-41e9-845f-861eb12cde93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:03:22.279551 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 951/1908] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:03:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:03:22.292769 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35287ec2-4f9c-4a96-bd53-aab2b6b644e1 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:03:22.294608 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-35287ec2-4f9c-4a96-bd53-aab2b6b644e1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:03:22.316775 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:03:22.355524 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35287ec2-4f9c-4a96-bd53-aab2b6b644e1 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:03:22.359310 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c382a10c-8dca-4786-ab4f-03c3b070919f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:03:22.359913 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c382a10c-8dca-4786-ab4f-03c3b070919f service neutron] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:03:22.360188 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c382a10c-8dca-4786-ab4f-03c3b070919f service neutron] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:03:22.360456 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c382a10c-8dca-4786-ab4f-03c3b070919f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:03:22.370119 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c382a10c-8dca-4786-ab4f-03c3b070919f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:03:22.371619 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c382a10c-8dca-4786-ab4f-03c3b070919f service neutron] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:03:22.381239 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 958/1909] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:03:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:03:25.164325 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc80d0ed-7ebe-4e02-8265-b358b7f0b970 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:03:25.164889 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc80d0ed-7ebe-4e02-8265-b358b7f0b970 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:03:25.164889 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc80d0ed-7ebe-4e02-8265-b358b7f0b970 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:03:25.166295 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 952/1910] 10.222.0.36 () {54 vars in 921 bytes} [Thu Oct 20 09:03:25 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 09:03:25.173923 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-549fa94a-f2ea-4cc6-b271-8901c8d2e372 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:03:25.174501 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-549fa94a-f2ea-4cc6-b271-8901c8d2e372 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:03:25.174831 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-549fa94a-f2ea-4cc6-b271-8901c8d2e372 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:03:25.204649 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-549fa94a-f2ea-4cc6-b271-8901c8d2e372 None None] MFA Rules not processed for user `a94e868115ec49918eda10724263930c`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:03:25.206219 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-549fa94a-f2ea-4cc6-b271-8901c8d2e372 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:03:25.294815 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 959/1911] 10.222.0.36 () {58 vars in 1018 bytes} [Thu Oct 20 09:03:25 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:03:25.304850 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc80d0ed-7ebe-4e02-8265-b358b7f0b970 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:03:25.307170 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-bc80d0ed-7ebe-4e02-8265-b358b7f0b970 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:03:25.353888 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:03:25.393172 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc80d0ed-7ebe-4e02-8265-b358b7f0b970 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:03:25.399231 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80694c29-d4b7-47ae-ae78-ccfd6bf03ad9 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce043914cd9042909f70ab0adb272d34', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:03:25.400126 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80694c29-d4b7-47ae-ae78-ccfd6bf03ad9 service glance] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:03:25.400417 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80694c29-d4b7-47ae-ae78-ccfd6bf03ad9 service glance] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:03:25.400696 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80694c29-d4b7-47ae-ae78-ccfd6bf03ad9 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:03:25.423709 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80694c29-d4b7-47ae-ae78-ccfd6bf03ad9 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:03:25.423709 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80694c29-d4b7-47ae-ae78-ccfd6bf03ad9 service glance] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:03:25.430795 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 953/1912] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:03:25 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 127 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:03:25.508319 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89aa70f7-bf91-461f-93de-d34fff80c9a0 None None] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:03:25.508695 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89aa70f7-bf91-461f-93de-d34fff80c9a0 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:03:25.509064 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89aa70f7-bf91-461f-93de-d34fff80c9a0 None None] PATH_INFO: `/` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:03:25.510943 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 960/1913] 10.222.0.36 () {54 vars in 921 bytes} [Thu Oct 20 09:03:25 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 09:03:25.517065 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7630692a-235c-4920-ba6b-0ab34df19abe None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:03:25.517421 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7630692a-235c-4920-ba6b-0ab34df19abe None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:03:25.517652 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7630692a-235c-4920-ba6b-0ab34df19abe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:03:25.539695 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-7630692a-235c-4920-ba6b-0ab34df19abe None None] MFA Rules not processed for user `794ca389002d41748d7925216ed964a5`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:03:25.541043 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-7630692a-235c-4920-ba6b-0ab34df19abe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:03:25.547961 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 954/1914] 10.222.0.36 () {58 vars in 1018 bytes} [Thu Oct 20 09:03:25 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:03:25.561320 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89aa70f7-bf91-461f-93de-d34fff80c9a0 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:03:25.575226 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89aa70f7-bf91-461f-93de-d34fff80c9a0 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:03:25.579550 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97dd40dd-faeb-434a-94d4-8ba011e94b76 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:03:25.580541 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97dd40dd-faeb-434a-94d4-8ba011e94b76 service neutron] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:03:25.581036 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97dd40dd-faeb-434a-94d4-8ba011e94b76 service neutron] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:03:25.581880 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97dd40dd-faeb-434a-94d4-8ba011e94b76 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:03:25.596085 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97dd40dd-faeb-434a-94d4-8ba011e94b76 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:03:25.597561 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97dd40dd-faeb-434a-94d4-8ba011e94b76 service neutron] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:03:25.613213 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 961/1915] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:03:25 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:03:25.751957 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b1c856-e95f-4ad5-a815-e94f7f6e5376 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:03:25.752638 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b1c856-e95f-4ad5-a815-e94f7f6e5376 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:03:25.753114 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b1c856-e95f-4ad5-a815-e94f7f6e5376 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:03:25.756095 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 955/1916] 10.222.0.36 () {54 vars in 921 bytes} [Thu Oct 20 09:03:25 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 09:03:25.764833 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d2ecec1-3871-4270-9436-6221b62702d4 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:03:25.765113 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d2ecec1-3871-4270-9436-6221b62702d4 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:03:25.765545 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d2ecec1-3871-4270-9436-6221b62702d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:03:25.790300 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-5d2ecec1-3871-4270-9436-6221b62702d4 None None] MFA Rules not processed for user `a94e868115ec49918eda10724263930c`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:03:25.792836 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-5d2ecec1-3871-4270-9436-6221b62702d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:03:25.805877 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 962/1917] 10.222.0.36 () {58 vars in 1018 bytes} [Thu Oct 20 09:03:25 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:03:25.817341 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82b1c856-e95f-4ad5-a815-e94f7f6e5376 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:03:25.833660 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82b1c856-e95f-4ad5-a815-e94f7f6e5376 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:03:25.840013 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45fa88e9-a344-4980-8a1a-bc6e7e2ae979 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:03:25.841528 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45fa88e9-a344-4980-8a1a-bc6e7e2ae979 service neutron] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:03:25.842066 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45fa88e9-a344-4980-8a1a-bc6e7e2ae979 service neutron] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:03:25.842879 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45fa88e9-a344-4980-8a1a-bc6e7e2ae979 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:03:25.865667 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45fa88e9-a344-4980-8a1a-bc6e7e2ae979 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:03:25.867569 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45fa88e9-a344-4980-8a1a-bc6e7e2ae979 service neutron] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:03:25.885614 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 956/1918] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:03:25 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 69 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 20 09:03:26.831985 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2543ea37-9335-4aa1-b2af-4c987a603592 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:03:26.832800 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2543ea37-9335-4aa1-b2af-4c987a603592 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:03:26.833306 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2543ea37-9335-4aa1-b2af-4c987a603592 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:03:26.863999 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-2543ea37-9335-4aa1-b2af-4c987a603592 None None] MFA Rules not processed for user `65ca2eae6db24bb094afaacdfb5c1980`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:03:26.865150 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-2543ea37-9335-4aa1-b2af-4c987a603592 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:03:26.939134 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 963/1919] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:03:26 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:03:26.943527 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45fa88e9-a344-4980-8a1a-bc6e7e2ae979 service neutron] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:03:26.958943 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45fa88e9-a344-4980-8a1a-bc6e7e2ae979 service neutron] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:03:26.965065 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55f8341c-946f-424b-8b52-3f776a3ccd41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:03:26.965937 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f8341c-946f-424b-8b52-3f776a3ccd41 admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:03:26.966212 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f8341c-946f-424b-8b52-3f776a3ccd41 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:03:26.966520 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f8341c-946f-424b-8b52-3f776a3ccd41 admin admin] PATH_INFO: `/v3/projects` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:03:26.967157 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55f8341c-946f-424b-8b52-3f776a3ccd41 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:03:26.969140 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55f8341c-946f-424b-8b52-3f776a3ccd41 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:03:27.000419 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 957/1920] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:03:26 2022] POST /identity/v3/projects => generated 379 bytes in 57 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 20 09:03:27.004584 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2543ea37-9335-4aa1-b2af-4c987a603592 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:03:27.015141 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2543ea37-9335-4aa1-b2af-4c987a603592 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:03:27.040116 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:03:27.080086 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7040767-71b0-4c3a-8c87-5e8a34ed6e06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:03:27.080965 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7040767-71b0-4c3a-8c87-5e8a34ed6e06 admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:03:27.081247 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7040767-71b0-4c3a-8c87-5e8a34ed6e06 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:03:27.081520 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7040767-71b0-4c3a-8c87-5e8a34ed6e06 admin admin] PATH_INFO: `/v3/users` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:03:27.082195 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7040767-71b0-4c3a-8c87-5e8a34ed6e06 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:03:27.084488 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7040767-71b0-4c3a-8c87-5e8a34ed6e06 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:03:27.111565 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 964/1921] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:03:27 2022] POST /identity/v3/users => generated 296 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:03:27.116462 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55f8341c-946f-424b-8b52-3f776a3ccd41 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:03:27.127283 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55f8341c-946f-424b-8b52-3f776a3ccd41 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:03:27.132250 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df30ce4b-22f5-4c54-9f6a-3725d2c455b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:03:27.133209 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df30ce4b-22f5-4c54-9f6a-3725d2c455b6 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:03:27.133663 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df30ce4b-22f5-4c54-9f6a-3725d2c455b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:03:27.134105 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df30ce4b-22f5-4c54-9f6a-3725d2c455b6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:03:27.135226 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df30ce4b-22f5-4c54-9f6a-3725d2c455b6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:03:27.138049 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df30ce4b-22f5-4c54-9f6a-3725d2c455b6 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:03:27.153301 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 958/1922] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:03:27 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:03:27.158919 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7040767-71b0-4c3a-8c87-5e8a34ed6e06 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:03:27.171107 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7040767-71b0-4c3a-8c87-5e8a34ed6e06 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:03:27.175790 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc0ca3de-2b89-4968-bcdd-5943a3826c8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:03:27.176439 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0ca3de-2b89-4968-bcdd-5943a3826c8a admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:03:27.176715 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0ca3de-2b89-4968-bcdd-5943a3826c8a admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:03:27.176987 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0ca3de-2b89-4968-bcdd-5943a3826c8a admin admin] PATH_INFO: `/v3/projects/630293d0e194481c86354df95acdd799/users/8b06169f2edd4818ab2382e76c3632ac/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:03:27.194986 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc0ca3de-2b89-4968-bcdd-5943a3826c8a admin admin] RBAC: Authorizing `identity:create_grant(project_id=630293d0e194481c86354df95acdd799, user_id=8b06169f2edd4818ab2382e76c3632ac, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:03:27.198073 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc0ca3de-2b89-4968-bcdd-5943a3826c8a admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:03:27.211930 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 965/1923] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:03:27 2022] PUT /identity/v3/projects/630293d0e194481c86354df95acdd799/users/8b06169f2edd4818ab2382e76c3632ac/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:03:27.216489 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df30ce4b-22f5-4c54-9f6a-3725d2c455b6 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:03:27.227034 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df30ce4b-22f5-4c54-9f6a-3725d2c455b6 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:03:27.288562 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ced61a6b-a21a-4291-a388-8a0169d36d08 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:03:27.289170 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced61a6b-a21a-4291-a388-8a0169d36d08 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:03:27.289446 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced61a6b-a21a-4291-a388-8a0169d36d08 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:03:27.289714 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced61a6b-a21a-4291-a388-8a0169d36d08 admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:03:27.290351 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ced61a6b-a21a-4291-a388-8a0169d36d08 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:03:27.291973 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ced61a6b-a21a-4291-a388-8a0169d36d08 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:03:27.302420 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 959/1924] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:03:27 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 20 09:03:27.307325 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc0ca3de-2b89-4968-bcdd-5943a3826c8a admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:03:27.318072 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc0ca3de-2b89-4968-bcdd-5943a3826c8a admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:03:27.323419 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b9a6174-2940-4fc2-abf1-f566e9880384 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:03:27.324080 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b9a6174-2940-4fc2-abf1-f566e9880384 admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:03:27.324362 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b9a6174-2940-4fc2-abf1-f566e9880384 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:03:27.324642 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b9a6174-2940-4fc2-abf1-f566e9880384 admin admin] PATH_INFO: `/v3/projects/630293d0e194481c86354df95acdd799/users/8b06169f2edd4818ab2382e76c3632ac/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:03:27.326746 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b9a6174-2940-4fc2-abf1-f566e9880384 admin admin] RBAC: Authorizing `identity:create_grant(project_id=630293d0e194481c86354df95acdd799, user_id=8b06169f2edd4818ab2382e76c3632ac, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:03:27.329452 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b9a6174-2940-4fc2-abf1-f566e9880384 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:03:27.340648 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 966/1925] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:03:27 2022] PUT /identity/v3/projects/630293d0e194481c86354df95acdd799/users/8b06169f2edd4818ab2382e76c3632ac/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:03:30.127565 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6db8a87-35b7-40e7-8e31-b7eff6622f3e None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:03:30.128172 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6db8a87-35b7-40e7-8e31-b7eff6622f3e None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:03:30.128293 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6db8a87-35b7-40e7-8e31-b7eff6622f3e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:03:30.165903 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-d6db8a87-35b7-40e7-8e31-b7eff6622f3e None None] MFA Rules not processed for user `7e0951446ad449d6b725fe6cc14df1f2`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:03:30.186546 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-d6db8a87-35b7-40e7-8e31-b7eff6622f3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:03:30.334353 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 960/1926] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:03:30 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 208 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:03:33.070444 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365cbca8-dbc4-4d67-a712-850c9b62b6bb None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:03:33.070939 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365cbca8-dbc4-4d67-a712-850c9b62b6bb None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:03:33.071150 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365cbca8-dbc4-4d67-a712-850c9b62b6bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:03:33.142308 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-365cbca8-dbc4-4d67-a712-850c9b62b6bb None None] MFA Rules not processed for user `8b06169f2edd4818ab2382e76c3632ac`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:03:33.143891 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-365cbca8-dbc4-4d67-a712-850c9b62b6bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:03:33.240013 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 967/1927] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:03:33 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 20 09:03:33.254402 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6db8a87-35b7-40e7-8e31-b7eff6622f3e None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:03:33.255952 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-d6db8a87-35b7-40e7-8e31-b7eff6622f3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:03:33.281980 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:03:33.350622 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6db8a87-35b7-40e7-8e31-b7eff6622f3e None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:03:33.357109 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5610ccb9-a364-4ba6-beae-f3ac5b3452b9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a94e868115ec49918eda10724263930c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:03:33.357938 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5610ccb9-a364-4ba6-beae-f3ac5b3452b9 service nova] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:03:33.358400 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5610ccb9-a364-4ba6-beae-f3ac5b3452b9 service nova] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:03:33.359372 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5610ccb9-a364-4ba6-beae-f3ac5b3452b9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:03:33.432829 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e5fc0f-6a07-4f43-8529-8e65ed61ec06 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:03:33.433358 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e5fc0f-6a07-4f43-8529-8e65ed61ec06 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:03:33.433804 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e5fc0f-6a07-4f43-8529-8e65ed61ec06 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:03:33.448849 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5610ccb9-a364-4ba6-beae-f3ac5b3452b9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:03:33.450496 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5610ccb9-a364-4ba6-beae-f3ac5b3452b9 service nova] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:03:33.458014 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-c0e5fc0f-6a07-4f43-8529-8e65ed61ec06 None None] MFA Rules not processed for user `f4b0e9187ce74f7888e9e7094be5c824`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:03:33.459544 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-c0e5fc0f-6a07-4f43-8529-8e65ed61ec06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:03:33.469802 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 961/1928] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:03:33 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 216 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:03:33.529171 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 968/1929] 10.222.0.36 () {58 vars in 1009 bytes} [Thu Oct 20 09:03:33 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:03:33.547167 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5610ccb9-a364-4ba6-beae-f3ac5b3452b9 service nova] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:03:33.548705 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-5610ccb9-a364-4ba6-beae-f3ac5b3452b9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:03:33.704028 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5610ccb9-a364-4ba6-beae-f3ac5b3452b9 service nova] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:03:33.726007 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-333682ea-65ad-4591-9535-1228ac833af7 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4d558918cd8b4ebea4fe8765297304db', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:03:33.728028 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-333682ea-65ad-4591-9535-1228ac833af7 service swift] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:03:33.728689 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-333682ea-65ad-4591-9535-1228ac833af7 service swift] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:03:33.728965 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-333682ea-65ad-4591-9535-1228ac833af7 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:03:33.739559 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-333682ea-65ad-4591-9535-1228ac833af7 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:03:33.741909 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-333682ea-65ad-4591-9535-1228ac833af7 service swift] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:03:33.752319 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 962/1930] 10.222.0.36 () {60 vars in 1384 bytes} [Thu Oct 20 09:03:33 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 206 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 20 09:03:43.519704 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=92,keystone:INSERT=5 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:03:43.748156 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=79 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:03:48.817842 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e5fc0f-6a07-4f43-8529-8e65ed61ec06 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:03:48.819253 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-c0e5fc0f-6a07-4f43-8529-8e65ed61ec06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:03:48.896506 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e5fc0f-6a07-4f43-8529-8e65ed61ec06 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:03:48.901362 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb07f2a9-4d13-4172-a4c6-d56a8d3faeef service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ad03ab76b9c44b1cb369475afa25eddd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:03:48.902014 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb07f2a9-4d13-4172-a4c6-d56a8d3faeef service placement] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:03:48.902297 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb07f2a9-4d13-4172-a4c6-d56a8d3faeef service placement] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:03:48.902594 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb07f2a9-4d13-4172-a4c6-d56a8d3faeef service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:03:48.904274 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-cb07f2a9-4d13-4172-a4c6-d56a8d3faeef service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:03:48.915384 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb07f2a9-4d13-4172-a4c6-d56a8d3faeef service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:03:48.916960 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb07f2a9-4d13-4172-a4c6-d56a8d3faeef service placement] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:03:48.926562 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 969/1931] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:03:48 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:03:54.522538 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b2bfad6-da4f-497d-9c0e-6dddebb58334 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:03:54.523066 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b2bfad6-da4f-497d-9c0e-6dddebb58334 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:03:54.523066 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b2bfad6-da4f-497d-9c0e-6dddebb58334 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:03:54.542350 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-6b2bfad6-da4f-497d-9c0e-6dddebb58334 None None] MFA Rules not processed for user `65ca2eae6db24bb094afaacdfb5c1980`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:03:54.543698 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-6b2bfad6-da4f-497d-9c0e-6dddebb58334 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:03:54.550694 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 963/1932] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:03:54 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:03:57.934707 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff66e76-6fb0-481d-a350-19de036e7cf8 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:03:57.935182 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff66e76-6fb0-481d-a350-19de036e7cf8 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:03:57.935297 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff66e76-6fb0-481d-a350-19de036e7cf8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:03:57.954414 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-bff66e76-6fb0-481d-a350-19de036e7cf8 None None] MFA Rules not processed for user `f4b0e9187ce74f7888e9e7094be5c824`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:03:57.955737 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-bff66e76-6fb0-481d-a350-19de036e7cf8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:03:57.963470 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 970/1933] 10.222.0.36 () {58 vars in 967 bytes} [Thu Oct 20 09:03:57 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:03:57.985402 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b2bfad6-da4f-497d-9c0e-6dddebb58334 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:03:57.996968 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b2bfad6-da4f-497d-9c0e-6dddebb58334 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:03:58.001805 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dde1c50f-d8e4-485a-a01e-936fdfc81524 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4d558918cd8b4ebea4fe8765297304db', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:03:58.002399 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde1c50f-d8e4-485a-a01e-936fdfc81524 service swift] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:03:58.002704 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde1c50f-d8e4-485a-a01e-936fdfc81524 service swift] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:03:58.002975 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde1c50f-d8e4-485a-a01e-936fdfc81524 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:03:58.010784 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dde1c50f-d8e4-485a-a01e-936fdfc81524 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:03:58.012354 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dde1c50f-d8e4-485a-a01e-936fdfc81524 service swift] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:03:58.020176 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 964/1934] 10.222.0.36 () {60 vars in 1384 bytes} [Thu Oct 20 09:03:57 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 20 09:04:07.949139 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=35 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:04:08.016966 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=9 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:04:23.965867 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff66e76-6fb0-481d-a350-19de036e7cf8 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:04:23.967299 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-bff66e76-6fb0-481d-a350-19de036e7cf8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:04:24.024337 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff66e76-6fb0-481d-a350-19de036e7cf8 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:04:24.028348 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d59dc9b1-e6fa-4b42-a2e1-0faeac74bb00 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:04:24.028953 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d59dc9b1-e6fa-4b42-a2e1-0faeac74bb00 service neutron] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:04:24.029230 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d59dc9b1-e6fa-4b42-a2e1-0faeac74bb00 service neutron] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:04:24.029501 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d59dc9b1-e6fa-4b42-a2e1-0faeac74bb00 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:04:24.038352 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d59dc9b1-e6fa-4b42-a2e1-0faeac74bb00 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:04:24.039827 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d59dc9b1-e6fa-4b42-a2e1-0faeac74bb00 service neutron] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:04:24.049748 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 971/1935] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:04:23 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:04:31.966709 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7858d626-b3da-446a-b24f-9deeb9b6c868 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:04:31.967183 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7858d626-b3da-446a-b24f-9deeb9b6c868 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:04:31.967289 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7858d626-b3da-446a-b24f-9deeb9b6c868 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:04:31.987789 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-7858d626-b3da-446a-b24f-9deeb9b6c868 None None] MFA Rules not processed for user `8002ab26f6d64822aaa65aa730781b76`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:04:31.988995 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-7858d626-b3da-446a-b24f-9deeb9b6c868 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:04:32.042461 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 965/1936] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:04:31 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:04:34.045532 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=26 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:04:42.035985 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=25 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:04:44.817823 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e38a594d-4df8-4fbf-9c2a-f8cfff8d7522 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:04:44.818284 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e38a594d-4df8-4fbf-9c2a-f8cfff8d7522 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:04:44.818388 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e38a594d-4df8-4fbf-9c2a-f8cfff8d7522 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:04:44.838438 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-e38a594d-4df8-4fbf-9c2a-f8cfff8d7522 None None] MFA Rules not processed for user `7f932695a5994c5790d14c210ea65326`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:04:44.840521 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-e38a594d-4df8-4fbf-9c2a-f8cfff8d7522 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:04:44.895297 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 972/1937] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:04:44 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:04:52.324130 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7858d626-b3da-446a-b24f-9deeb9b6c868 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:04:52.325779 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-7858d626-b3da-446a-b24f-9deeb9b6c868 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:04:52.405775 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7858d626-b3da-446a-b24f-9deeb9b6c868 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:04:52.410986 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-014d6614-8454-4e18-802f-aa5f1598588b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:04:52.411628 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-014d6614-8454-4e18-802f-aa5f1598588b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:04:52.411915 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-014d6614-8454-4e18-802f-aa5f1598588b admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:04:52.412249 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-014d6614-8454-4e18-802f-aa5f1598588b admin admin] PATH_INFO: `/v3/users/7e0951446ad449d6b725fe6cc14df1f2` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:04:52.413738 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-014d6614-8454-4e18-802f-aa5f1598588b admin admin] RBAC: Authorizing `identity:delete_user(user_id=7e0951446ad449d6b725fe6cc14df1f2)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:04:52.415886 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-014d6614-8454-4e18-802f-aa5f1598588b admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:04:52.472764 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-014d6614-8454-4e18-802f-aa5f1598588b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7e0951446ad449d6b725fe6cc14df1f2'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:04:52.483218 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-014d6614-8454-4e18-802f-aa5f1598588b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7e0951446ad449d6b725fe6cc14df1f2'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:04:52.501792 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-014d6614-8454-4e18-802f-aa5f1598588b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7e0951446ad449d6b725fe6cc14df1f2'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:04:52.511765 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-014d6614-8454-4e18-802f-aa5f1598588b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7e0951446ad449d6b725fe6cc14df1f2'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:04:52.511924 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-014d6614-8454-4e18-802f-aa5f1598588b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7e0951446ad449d6b725fe6cc14df1f2'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:04:52.518448 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 966/1938] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:04:52 2022] DELETE /identity/v3/users/7e0951446ad449d6b725fe6cc14df1f2 => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:04:52.896588 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e38a594d-4df8-4fbf-9c2a-f8cfff8d7522 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:04:52.920691 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e38a594d-4df8-4fbf-9c2a-f8cfff8d7522 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:04:52.984048 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f25ffe-cb6b-40c1-9e69-2be72a3227b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:04:52.984686 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f25ffe-cb6b-40c1-9e69-2be72a3227b3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:04:52.984963 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f25ffe-cb6b-40c1-9e69-2be72a3227b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:04:52.985231 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f25ffe-cb6b-40c1-9e69-2be72a3227b3 admin admin] PATH_INFO: `/v3/projects/9dc1a3b80bd34c8e8f691a9b47457f35` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:04:52.986356 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4f25ffe-cb6b-40c1-9e69-2be72a3227b3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9dc1a3b80bd34c8e8f691a9b47457f35)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:04:52.988396 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4f25ffe-cb6b-40c1-9e69-2be72a3227b3 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:04:53.040506 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-f4f25ffe-cb6b-40c1-9e69-2be72a3227b3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9dc1a3b80bd34c8e8f691a9b47457f35', 'request_id': 'req-f4f25ffe-cb6b-40c1-9e69-2be72a3227b3'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:04:53.047159 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-f4f25ffe-cb6b-40c1-9e69-2be72a3227b3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9dc1a3b80bd34c8e8f691a9b47457f35', 'request_id': 'req-f4f25ffe-cb6b-40c1-9e69-2be72a3227b3'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:04:53.052700 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-f4f25ffe-cb6b-40c1-9e69-2be72a3227b3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9dc1a3b80bd34c8e8f691a9b47457f35', 'request_id': 'req-f4f25ffe-cb6b-40c1-9e69-2be72a3227b3'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:04:53.066129 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-f4f25ffe-cb6b-40c1-9e69-2be72a3227b3 admin admin] The token cache is being invalidate because project 9dc1a3b80bd34c8e8f691a9b47457f35 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81561) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:04:53.066410 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-f4f25ffe-cb6b-40c1-9e69-2be72a3227b3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:04:53.067150 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-f4f25ffe-cb6b-40c1-9e69-2be72a3227b3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:04:53.069038 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 973/1939] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:04:52 2022] DELETE /identity/v3/projects/9dc1a3b80bd34c8e8f691a9b47457f35 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:04:53.073645 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f921e9-9473-4a87-bf57-8f42856a0fed None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:04:53.073924 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f921e9-9473-4a87-bf57-8f42856a0fed None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:04:53.074188 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f921e9-9473-4a87-bf57-8f42856a0fed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:04:53.096347 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-84f921e9-9473-4a87-bf57-8f42856a0fed None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:04:53.097709 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-84f921e9-9473-4a87-bf57-8f42856a0fed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:04:53.164159 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 967/1940] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:04:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:04:53.172910 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ed4ee8-ba96-46d5-ba0e-208f8184de2e None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:04:53.173122 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ed4ee8-ba96-46d5-ba0e-208f8184de2e None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:04:53.173397 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ed4ee8-ba96-46d5-ba0e-208f8184de2e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:04:53.193899 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-94ed4ee8-ba96-46d5-ba0e-208f8184de2e None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:04:53.195143 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-94ed4ee8-ba96-46d5-ba0e-208f8184de2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:04:53.203703 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 974/1941] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:04:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:04:53.208169 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84f921e9-9473-4a87-bf57-8f42856a0fed None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:04:53.218072 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84f921e9-9473-4a87-bf57-8f42856a0fed None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:04:53.222613 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddfbaf8f-813b-41ec-bc72-9a0c07abdf8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:04:53.223206 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddfbaf8f-813b-41ec-bc72-9a0c07abdf8a admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:04:53.223482 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddfbaf8f-813b-41ec-bc72-9a0c07abdf8a admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:04:53.223751 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddfbaf8f-813b-41ec-bc72-9a0c07abdf8a admin admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:04:53.224418 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddfbaf8f-813b-41ec-bc72-9a0c07abdf8a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 09:04:53.224716 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddfbaf8f-813b-41ec-bc72-9a0c07abdf8a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:04:53.226310 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddfbaf8f-813b-41ec-bc72-9a0c07abdf8a admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:04:53.237964 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 968/1942] 10.222.0.36 () {58 vars in 1153 bytes} [Thu Oct 20 09:04:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 09:04:53.242043 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94ed4ee8-ba96-46d5-ba0e-208f8184de2e None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:04:53.251710 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94ed4ee8-ba96-46d5-ba0e-208f8184de2e None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:04:53.256331 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0380cb1e-1966-4553-af56-731448433491 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:04:53.256914 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0380cb1e-1966-4553-af56-731448433491 admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:04:53.257195 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0380cb1e-1966-4553-af56-731448433491 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:04:53.257467 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0380cb1e-1966-4553-af56-731448433491 admin admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:04:53.258100 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0380cb1e-1966-4553-af56-731448433491 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:04:53.260108 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0380cb1e-1966-4553-af56-731448433491 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:04:53.289718 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 975/1943] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:04:53 2022] POST /identity/v3/projects => generated 399 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:04:53.292116 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddfbaf8f-813b-41ec-bc72-9a0c07abdf8a admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:04:53.302397 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddfbaf8f-813b-41ec-bc72-9a0c07abdf8a admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:04:53.305928 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:04:53.361586 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0a53c5a-f6c5-44a7-b8d5-d8319b69822a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:04:53.362214 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a53c5a-f6c5-44a7-b8d5-d8319b69822a admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:04:53.362450 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a53c5a-f6c5-44a7-b8d5-d8319b69822a admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:04:53.362747 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a53c5a-f6c5-44a7-b8d5-d8319b69822a admin admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:04:53.363381 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0a53c5a-f6c5-44a7-b8d5-d8319b69822a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:04:53.365303 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0a53c5a-f6c5-44a7-b8d5-d8319b69822a admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:04:53.388006 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 969/1944] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:04:53 2022] POST /identity/v3/users => generated 300 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:04:53.392278 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0380cb1e-1966-4553-af56-731448433491 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:04:53.402679 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0380cb1e-1966-4553-af56-731448433491 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:04:53.407295 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42952431-c3d8-4f19-b2c8-0f9a2c5c8d60 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:04:53.407902 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42952431-c3d8-4f19-b2c8-0f9a2c5c8d60 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:04:53.408280 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42952431-c3d8-4f19-b2c8-0f9a2c5c8d60 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:04:53.408553 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42952431-c3d8-4f19-b2c8-0f9a2c5c8d60 admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:04:53.409182 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42952431-c3d8-4f19-b2c8-0f9a2c5c8d60 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:04:53.410809 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42952431-c3d8-4f19-b2c8-0f9a2c5c8d60 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:04:53.412066 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:04:53.424069 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 976/1945] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:04:53 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:04:53.425915 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0a53c5a-f6c5-44a7-b8d5-d8319b69822a admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:04:53.436249 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0a53c5a-f6c5-44a7-b8d5-d8319b69822a admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:04:53.440850 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e51fb1c-0dbe-4d09-93ae-813f3aa530cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:04:53.441435 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e51fb1c-0dbe-4d09-93ae-813f3aa530cd admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:04:53.441712 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e51fb1c-0dbe-4d09-93ae-813f3aa530cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:04:53.441983 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e51fb1c-0dbe-4d09-93ae-813f3aa530cd admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:04:53.442661 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e51fb1c-0dbe-4d09-93ae-813f3aa530cd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:04:53.444275 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e51fb1c-0dbe-4d09-93ae-813f3aa530cd admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:04:53.455238 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 970/1946] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:04:53 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:04:53.459322 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42952431-c3d8-4f19-b2c8-0f9a2c5c8d60 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:04:53.469578 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42952431-c3d8-4f19-b2c8-0f9a2c5c8d60 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:04:53.474154 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8accd4d-7a9c-4519-8bfc-fe4e43d94818 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:04:53.474831 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8accd4d-7a9c-4519-8bfc-fe4e43d94818 admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:04:53.475108 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8accd4d-7a9c-4519-8bfc-fe4e43d94818 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:04:53.475376 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8accd4d-7a9c-4519-8bfc-fe4e43d94818 admin admin] PATH_INFO: `/v3/projects/362d73d41e7e4fd28099fa22a255b171/users/338a5f87cb3445b1827ad1193e60d6e5/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:04:53.489735 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8accd4d-7a9c-4519-8bfc-fe4e43d94818 admin admin] RBAC: Authorizing `identity:create_grant(project_id=362d73d41e7e4fd28099fa22a255b171, user_id=338a5f87cb3445b1827ad1193e60d6e5, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:04:53.492485 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8accd4d-7a9c-4519-8bfc-fe4e43d94818 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:04:53.505882 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 977/1947] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:04:53 2022] PUT /identity/v3/projects/362d73d41e7e4fd28099fa22a255b171/users/338a5f87cb3445b1827ad1193e60d6e5/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:04:53.516609 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6624bd-0aab-4828-997c-a5c36c013005 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:04:53.516984 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6624bd-0aab-4828-997c-a5c36c013005 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:04:53.517319 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6624bd-0aab-4828-997c-a5c36c013005 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:04:53.544506 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-ff6624bd-0aab-4828-997c-a5c36c013005 None None] MFA Rules not processed for user `338a5f87cb3445b1827ad1193e60d6e5`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:04:53.545705 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-ff6624bd-0aab-4828-997c-a5c36c013005 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:04:53.608276 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 971/1948] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:04:53 2022] POST /identity/v3/auth/tokens => generated 3276 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:04:53.619617 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8accd4d-7a9c-4519-8bfc-fe4e43d94818 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:04:53.621346 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-a8accd4d-7a9c-4519-8bfc-fe4e43d94818 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:04:53.677154 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8accd4d-7a9c-4519-8bfc-fe4e43d94818 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:04:53.680683 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1ea966c-0a55-4297-ba20-d1bd2800a54c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:04:53.681280 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1ea966c-0a55-4297-ba20-d1bd2800a54c service cinder] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:04:53.681554 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1ea966c-0a55-4297-ba20-d1bd2800a54c service cinder] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:04:53.681821 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1ea966c-0a55-4297-ba20-d1bd2800a54c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:04:53.690525 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1ea966c-0a55-4297-ba20-d1bd2800a54c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:04:53.691955 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1ea966c-0a55-4297-ba20-d1bd2800a54c service cinder] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:04:53.700904 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 978/1949] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:04:53 2022] GET /identity/v3/auth/tokens => generated 3276 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:05:00.880673 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6624bd-0aab-4828-997c-a5c36c013005 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:05:00.941812 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6624bd-0aab-4828-997c-a5c36c013005 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:05:00.946251 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d04e9eeb-a606-433b-a043-3f73a009eedf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:05:00.946900 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d04e9eeb-a606-433b-a043-3f73a009eedf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:05:00.947180 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d04e9eeb-a606-433b-a043-3f73a009eedf admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:05:00.947449 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d04e9eeb-a606-433b-a043-3f73a009eedf admin admin] PATH_INFO: `/v3/users/338a5f87cb3445b1827ad1193e60d6e5` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:05:00.948539 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d04e9eeb-a606-433b-a043-3f73a009eedf admin admin] RBAC: Authorizing `identity:delete_user(user_id=338a5f87cb3445b1827ad1193e60d6e5)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:05:00.951113 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d04e9eeb-a606-433b-a043-3f73a009eedf admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:05:00.966363 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=96,keystone:INSERT=3,keystone:DELETE=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:05:01.002696 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-d04e9eeb-a606-433b-a043-3f73a009eedf admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '338a5f87cb3445b1827ad1193e60d6e5'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:05:01.009957 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-d04e9eeb-a606-433b-a043-3f73a009eedf admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '338a5f87cb3445b1827ad1193e60d6e5'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:05:01.029774 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-d04e9eeb-a606-433b-a043-3f73a009eedf admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '338a5f87cb3445b1827ad1193e60d6e5'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:05:01.038060 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-d04e9eeb-a606-433b-a043-3f73a009eedf admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '338a5f87cb3445b1827ad1193e60d6e5'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:05:01.039712 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-d04e9eeb-a606-433b-a043-3f73a009eedf admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '338a5f87cb3445b1827ad1193e60d6e5'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:05:01.046667 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 972/1950] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:05:00 2022] DELETE /identity/v3/users/338a5f87cb3445b1827ad1193e60d6e5 => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:05:01.051263 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-670979d0-c975-4fae-b947-d21b7354a0d1 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:05:01.051746 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-670979d0-c975-4fae-b947-d21b7354a0d1 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:05:01.051921 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-670979d0-c975-4fae-b947-d21b7354a0d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:05:01.072405 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-670979d0-c975-4fae-b947-d21b7354a0d1 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:05:01.073600 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-670979d0-c975-4fae-b947-d21b7354a0d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:05:01.138858 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 979/1951] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:05:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:05:01.148660 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d04e9eeb-a606-433b-a043-3f73a009eedf admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:05:01.150245 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-d04e9eeb-a606-433b-a043-3f73a009eedf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:05:01.213622 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d04e9eeb-a606-433b-a043-3f73a009eedf admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:05:01.217913 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7e5845b-0a11-430d-a534-3bff2580ec92 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:05:01.218542 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e5845b-0a11-430d-a534-3bff2580ec92 service neutron] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:05:01.218821 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e5845b-0a11-430d-a534-3bff2580ec92 service neutron] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:05:01.219090 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e5845b-0a11-430d-a534-3bff2580ec92 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:05:01.228838 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7e5845b-0a11-430d-a534-3bff2580ec92 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:05:01.230316 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7e5845b-0a11-430d-a534-3bff2580ec92 service neutron] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:05:01.240561 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 973/1952] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:05:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:05:01.581683 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-670979d0-c975-4fae-b947-d21b7354a0d1 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:05:01.583148 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-670979d0-c975-4fae-b947-d21b7354a0d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:05:01.594853 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-670979d0-c975-4fae-b947-d21b7354a0d1 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:05:01.599344 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93af6197-ffe0-41e5-ba94-c3d61bb39fd0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:05:01.599966 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93af6197-ffe0-41e5-ba94-c3d61bb39fd0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:05:01.600246 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93af6197-ffe0-41e5-ba94-c3d61bb39fd0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:05:01.600528 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93af6197-ffe0-41e5-ba94-c3d61bb39fd0 admin admin] PATH_INFO: `/v3/projects/362d73d41e7e4fd28099fa22a255b171` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:05:01.601638 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93af6197-ffe0-41e5-ba94-c3d61bb39fd0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=362d73d41e7e4fd28099fa22a255b171)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:05:01.603577 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93af6197-ffe0-41e5-ba94-c3d61bb39fd0 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:05:01.651849 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-93af6197-ffe0-41e5-ba94-c3d61bb39fd0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '362d73d41e7e4fd28099fa22a255b171', 'request_id': 'req-93af6197-ffe0-41e5-ba94-c3d61bb39fd0'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:05:01.657993 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-93af6197-ffe0-41e5-ba94-c3d61bb39fd0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '362d73d41e7e4fd28099fa22a255b171', 'request_id': 'req-93af6197-ffe0-41e5-ba94-c3d61bb39fd0'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:05:01.663461 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-93af6197-ffe0-41e5-ba94-c3d61bb39fd0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '362d73d41e7e4fd28099fa22a255b171', 'request_id': 'req-93af6197-ffe0-41e5-ba94-c3d61bb39fd0'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:05:01.671207 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:05:01.677048 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-93af6197-ffe0-41e5-ba94-c3d61bb39fd0 admin admin] The token cache is being invalidate because project 362d73d41e7e4fd28099fa22a255b171 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81561) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:05:01.677629 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-93af6197-ffe0-41e5-ba94-c3d61bb39fd0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:05:01.678659 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-93af6197-ffe0-41e5-ba94-c3d61bb39fd0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:05:01.680439 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 980/1953] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:05:01 2022] DELETE /identity/v3/projects/362d73d41e7e4fd28099fa22a255b171 => generated 0 bytes in 99 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:05:01.686781 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-989ca443-019b-4576-9aca-ba3d8153c9e2 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:05:01.687136 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-989ca443-019b-4576-9aca-ba3d8153c9e2 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:05:01.687466 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-989ca443-019b-4576-9aca-ba3d8153c9e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:05:01.707071 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-989ca443-019b-4576-9aca-ba3d8153c9e2 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:05:01.708249 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-989ca443-019b-4576-9aca-ba3d8153c9e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:05:01.766319 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 974/1954] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:05:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 20 09:05:01.774123 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de3173f-7b3f-4011-bcb0-ff841d51b9b9 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:05:01.774407 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de3173f-7b3f-4011-bcb0-ff841d51b9b9 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:05:01.774721 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de3173f-7b3f-4011-bcb0-ff841d51b9b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:05:01.793829 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-9de3173f-7b3f-4011-bcb0-ff841d51b9b9 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:05:01.795046 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-9de3173f-7b3f-4011-bcb0-ff841d51b9b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:05:01.802933 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 981/1955] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:05:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:05:01.806885 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-989ca443-019b-4576-9aca-ba3d8153c9e2 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:05:01.816945 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-989ca443-019b-4576-9aca-ba3d8153c9e2 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:05:01.821481 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a68699e-b319-4e96-a67a-22562a7cc1a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:05:01.822068 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a68699e-b319-4e96-a67a-22562a7cc1a0 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:05:01.822346 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a68699e-b319-4e96-a67a-22562a7cc1a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:05:01.822658 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a68699e-b319-4e96-a67a-22562a7cc1a0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:05:01.823300 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a68699e-b319-4e96-a67a-22562a7cc1a0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 09:05:01.823603 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a68699e-b319-4e96-a67a-22562a7cc1a0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:05:01.825772 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a68699e-b319-4e96-a67a-22562a7cc1a0 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:05:01.837132 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 975/1956] 10.222.0.36 () {58 vars in 1153 bytes} [Thu Oct 20 09:05:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 09:05:01.841395 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de3173f-7b3f-4011-bcb0-ff841d51b9b9 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:05:01.850498 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de3173f-7b3f-4011-bcb0-ff841d51b9b9 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:05:01.855066 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3478e87d-903b-49b9-b787-0d2488ef98f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:05:01.856214 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3478e87d-903b-49b9-b787-0d2488ef98f9 admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:05:01.856489 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3478e87d-903b-49b9-b787-0d2488ef98f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:05:01.856758 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3478e87d-903b-49b9-b787-0d2488ef98f9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:05:01.857387 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3478e87d-903b-49b9-b787-0d2488ef98f9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:05:01.859323 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3478e87d-903b-49b9-b787-0d2488ef98f9 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:05:01.886289 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 982/1957] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:05:01 2022] POST /identity/v3/projects => generated 385 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:05:01.890261 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a68699e-b319-4e96-a67a-22562a7cc1a0 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:05:01.900061 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a68699e-b319-4e96-a67a-22562a7cc1a0 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:05:01.920954 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:DELETE=9,keystone:SELECT=90,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:05:01.957878 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8a74869-2ecd-45e9-b43b-d81caeb1d3f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:05:01.958497 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a74869-2ecd-45e9-b43b-d81caeb1d3f6 admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:05:01.958780 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a74869-2ecd-45e9-b43b-d81caeb1d3f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:05:01.959049 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a74869-2ecd-45e9-b43b-d81caeb1d3f6 admin admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:05:01.959670 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8a74869-2ecd-45e9-b43b-d81caeb1d3f6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:05:01.961576 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8a74869-2ecd-45e9-b43b-d81caeb1d3f6 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:05:01.982313 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 976/1958] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:05:01 2022] POST /identity/v3/users => generated 293 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 20 09:05:01.986455 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3478e87d-903b-49b9-b787-0d2488ef98f9 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:05:01.997150 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3478e87d-903b-49b9-b787-0d2488ef98f9 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:05:02.001634 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0c0dc00-8b6a-4111-8550-cb381e3b83d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:05:02.002214 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c0dc00-8b6a-4111-8550-cb381e3b83d8 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:05:02.002508 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c0dc00-8b6a-4111-8550-cb381e3b83d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:05:02.002781 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c0dc00-8b6a-4111-8550-cb381e3b83d8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:05:02.003411 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0c0dc00-8b6a-4111-8550-cb381e3b83d8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:05:02.004996 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0c0dc00-8b6a-4111-8550-cb381e3b83d8 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:05:02.015217 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 983/1959] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:05:01 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:05:02.020334 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8a74869-2ecd-45e9-b43b-d81caeb1d3f6 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:05:02.031454 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8a74869-2ecd-45e9-b43b-d81caeb1d3f6 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:05:02.036037 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c11fef7a-0812-467c-adf9-ea2b97b59d85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:05:02.036624 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c11fef7a-0812-467c-adf9-ea2b97b59d85 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:05:02.036902 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c11fef7a-0812-467c-adf9-ea2b97b59d85 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:05:02.037179 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c11fef7a-0812-467c-adf9-ea2b97b59d85 admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:05:02.037818 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c11fef7a-0812-467c-adf9-ea2b97b59d85 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:05:02.039459 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c11fef7a-0812-467c-adf9-ea2b97b59d85 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:05:02.050662 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 977/1960] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:05:02 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:05:02.054651 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0c0dc00-8b6a-4111-8550-cb381e3b83d8 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:05:02.065290 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0c0dc00-8b6a-4111-8550-cb381e3b83d8 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:05:02.069849 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed17651b-bb16-4836-838b-b2c4c87d2c13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:05:02.070509 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed17651b-bb16-4836-838b-b2c4c87d2c13 admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:05:02.070793 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed17651b-bb16-4836-838b-b2c4c87d2c13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:05:02.071067 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed17651b-bb16-4836-838b-b2c4c87d2c13 admin admin] PATH_INFO: `/v3/projects/020d4019643f4499b5162717fbf7b5b5/users/14ddd15ab87a4d0fbea1d6dfa910f8ce/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:05:02.085316 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed17651b-bb16-4836-838b-b2c4c87d2c13 admin admin] RBAC: Authorizing `identity:create_grant(project_id=020d4019643f4499b5162717fbf7b5b5, user_id=14ddd15ab87a4d0fbea1d6dfa910f8ce, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:05:02.088039 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed17651b-bb16-4836-838b-b2c4c87d2c13 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:05:02.101313 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 984/1961] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:05:02 2022] PUT /identity/v3/projects/020d4019643f4499b5162717fbf7b5b5/users/14ddd15ab87a4d0fbea1d6dfa910f8ce/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:05:02.106839 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b15dd00-274e-44d4-a090-f1c311be3c3a None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:05:02.107113 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b15dd00-274e-44d4-a090-f1c311be3c3a None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:05:02.107384 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b15dd00-274e-44d4-a090-f1c311be3c3a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:05:02.126649 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-7b15dd00-274e-44d4-a090-f1c311be3c3a None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:05:02.127909 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-7b15dd00-274e-44d4-a090-f1c311be3c3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:05:02.185818 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 978/1962] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:05:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:05:02.195415 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed17651b-bb16-4836-838b-b2c4c87d2c13 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:05:02.197111 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-ed17651b-bb16-4836-838b-b2c4c87d2c13 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:05:02.253313 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed17651b-bb16-4836-838b-b2c4c87d2c13 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:05:02.256899 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4a9bad8-e36b-41a8-9be7-c67f59d271af service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:05:02.257488 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4a9bad8-e36b-41a8-9be7-c67f59d271af service neutron] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:05:02.257761 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4a9bad8-e36b-41a8-9be7-c67f59d271af service neutron] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:05:02.258026 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4a9bad8-e36b-41a8-9be7-c67f59d271af service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:05:02.266968 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4a9bad8-e36b-41a8-9be7-c67f59d271af service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:05:02.268393 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4a9bad8-e36b-41a8-9be7-c67f59d271af service neutron] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:05:02.278182 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 985/1963] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:05:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:05:07.494995 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1397a64-d24c-40a3-bff7-5a1fa582854b None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:05:07.495471 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1397a64-d24c-40a3-bff7-5a1fa582854b None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:05:07.495627 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1397a64-d24c-40a3-bff7-5a1fa582854b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:05:07.531760 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-d1397a64-d24c-40a3-bff7-5a1fa582854b None None] MFA Rules not processed for user `14ddd15ab87a4d0fbea1d6dfa910f8ce`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:05:07.533071 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-d1397a64-d24c-40a3-bff7-5a1fa582854b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:05:07.624638 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 979/1964] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:05:07 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:05:07.629071 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4a9bad8-e36b-41a8-9be7-c67f59d271af service neutron] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:05:07.641207 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4a9bad8-e36b-41a8-9be7-c67f59d271af service neutron] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:05:07.647498 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6168ebc-21bb-4ff4-ae1b-2ed7ff3d1797 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:05:07.648892 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6168ebc-21bb-4ff4-ae1b-2ed7ff3d1797 admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:05:07.649346 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6168ebc-21bb-4ff4-ae1b-2ed7ff3d1797 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:05:07.649815 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6168ebc-21bb-4ff4-ae1b-2ed7ff3d1797 admin admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:05:07.650661 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6168ebc-21bb-4ff4-ae1b-2ed7ff3d1797 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:05:07.653296 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6168ebc-21bb-4ff4-ae1b-2ed7ff3d1797 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:05:07.697781 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 986/1965] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:05:07 2022] POST /identity/v3/projects => generated 385 bytes in 69 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:05:07.702107 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1397a64-d24c-40a3-bff7-5a1fa582854b None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:05:07.714180 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1397a64-d24c-40a3-bff7-5a1fa582854b None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:05:07.756027 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:05:07.794655 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1ca7757-5309-4e0c-af57-93480f2f3b42 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:05:07.795452 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1ca7757-5309-4e0c-af57-93480f2f3b42 admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:05:07.795879 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1ca7757-5309-4e0c-af57-93480f2f3b42 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:05:07.796162 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1ca7757-5309-4e0c-af57-93480f2f3b42 admin admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:05:07.796811 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1ca7757-5309-4e0c-af57-93480f2f3b42 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:05:07.799012 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1ca7757-5309-4e0c-af57-93480f2f3b42 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:05:07.832320 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 980/1966] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:05:07 2022] POST /identity/v3/users => generated 299 bytes in 131 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:05:07.836201 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6168ebc-21bb-4ff4-ae1b-2ed7ff3d1797 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:05:07.849399 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6168ebc-21bb-4ff4-ae1b-2ed7ff3d1797 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:05:07.854900 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12b06dae-663e-4daf-bafe-de2b49ca6d75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:05:07.856081 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b06dae-663e-4daf-bafe-de2b49ca6d75 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:05:07.856595 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b06dae-663e-4daf-bafe-de2b49ca6d75 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:05:07.857285 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b06dae-663e-4daf-bafe-de2b49ca6d75 admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:05:07.858149 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12b06dae-663e-4daf-bafe-de2b49ca6d75 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:05:07.860015 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12b06dae-663e-4daf-bafe-de2b49ca6d75 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:05:07.878556 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 987/1967] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:05:07 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:05:07.878932 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1ca7757-5309-4e0c-af57-93480f2f3b42 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:05:07.890389 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1ca7757-5309-4e0c-af57-93480f2f3b42 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:05:07.895558 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5d1fdb5-51d2-4e06-babd-cc096d36d569 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:05:07.896240 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d1fdb5-51d2-4e06-babd-cc096d36d569 admin admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:05:07.896523 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d1fdb5-51d2-4e06-babd-cc096d36d569 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:05:07.896798 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d1fdb5-51d2-4e06-babd-cc096d36d569 admin admin] PATH_INFO: `/v3/projects/7e1c9e1c5a2249008cc4b4c7c12ff61e/users/69303fcf63304748b1988ce4a19317af/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:05:07.918288 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5d1fdb5-51d2-4e06-babd-cc096d36d569 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7e1c9e1c5a2249008cc4b4c7c12ff61e, user_id=69303fcf63304748b1988ce4a19317af, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:05:07.922884 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5d1fdb5-51d2-4e06-babd-cc096d36d569 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:05:07.940820 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 981/1968] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:05:07 2022] PUT /identity/v3/projects/7e1c9e1c5a2249008cc4b4c7c12ff61e/users/69303fcf63304748b1988ce4a19317af/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 62 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:05:07.944791 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12b06dae-663e-4daf-bafe-de2b49ca6d75 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:05:07.962340 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12b06dae-663e-4daf-bafe-de2b49ca6d75 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:05:08.048749 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ac63b7b-639e-443f-b913-7b3463bff60f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:05:08.049431 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac63b7b-639e-443f-b913-7b3463bff60f admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:05:08.049709 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac63b7b-639e-443f-b913-7b3463bff60f admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:05:08.049979 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac63b7b-639e-443f-b913-7b3463bff60f admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:05:08.050652 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ac63b7b-639e-443f-b913-7b3463bff60f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:05:08.052395 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ac63b7b-639e-443f-b913-7b3463bff60f admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:05:08.058934 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:05:08.065976 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 988/1969] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:05:07 2022] GET /identity/v3/roles => generated 1420 bytes in 121 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:05:08.069957 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5d1fdb5-51d2-4e06-babd-cc096d36d569 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:05:08.082347 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5d1fdb5-51d2-4e06-babd-cc096d36d569 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:05:08.087330 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d31bbf7-3791-41b3-a38a-d8abe4a0e8a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:05:08.087961 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d31bbf7-3791-41b3-a38a-d8abe4a0e8a4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:05:08.088306 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d31bbf7-3791-41b3-a38a-d8abe4a0e8a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:05:08.088990 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d31bbf7-3791-41b3-a38a-d8abe4a0e8a4 admin admin] PATH_INFO: `/v3/projects/7e1c9e1c5a2249008cc4b4c7c12ff61e/users/69303fcf63304748b1988ce4a19317af/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:05:08.091571 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d31bbf7-3791-41b3-a38a-d8abe4a0e8a4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7e1c9e1c5a2249008cc4b4c7c12ff61e, user_id=69303fcf63304748b1988ce4a19317af, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:05:08.094294 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d31bbf7-3791-41b3-a38a-d8abe4a0e8a4 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:05:08.105771 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 982/1970] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:05:08 2022] PUT /identity/v3/projects/7e1c9e1c5a2249008cc4b4c7c12ff61e/users/69303fcf63304748b1988ce4a19317af/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:05:08.719763 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d753acca-029f-4094-805b-c4ebf18f1c0b None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:05:08.720281 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d753acca-029f-4094-805b-c4ebf18f1c0b None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:05:08.720281 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d753acca-029f-4094-805b-c4ebf18f1c0b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:05:08.740146 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-d753acca-029f-4094-805b-c4ebf18f1c0b None None] MFA Rules not processed for user `fd9733d1bb2b4520b70090542615ddf9`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:05:08.741268 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-d753acca-029f-4094-805b-c4ebf18f1c0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:05:08.803608 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 989/1971] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:05:08 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:05:14.470554 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3c6f24-0344-486e-9d5f-685056896d6c None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:05:14.471396 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3c6f24-0344-486e-9d5f-685056896d6c None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:05:14.471665 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3c6f24-0344-486e-9d5f-685056896d6c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:05:14.502261 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-ea3c6f24-0344-486e-9d5f-685056896d6c None None] MFA Rules not processed for user `69303fcf63304748b1988ce4a19317af`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:05:14.503472 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-ea3c6f24-0344-486e-9d5f-685056896d6c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:05:14.579750 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 983/1972] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:05:14 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:05:14.592390 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d753acca-029f-4094-805b-c4ebf18f1c0b None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:05:14.593757 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-d753acca-029f-4094-805b-c4ebf18f1c0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:05:14.656879 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d753acca-029f-4094-805b-c4ebf18f1c0b None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:05:14.660405 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-366ffe1f-771c-4352-9289-3fed4c5ef29c service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce043914cd9042909f70ab0adb272d34', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:05:14.661012 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-366ffe1f-771c-4352-9289-3fed4c5ef29c service glance] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:05:14.661287 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-366ffe1f-771c-4352-9289-3fed4c5ef29c service glance] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:05:14.661550 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-366ffe1f-771c-4352-9289-3fed4c5ef29c service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:05:14.730902 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-366ffe1f-771c-4352-9289-3fed4c5ef29c service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:05:14.732389 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-366ffe1f-771c-4352-9289-3fed4c5ef29c service glance] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:05:14.743376 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 990/1973] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:05:14 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 151 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:05:14.903352 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f9c15fc-47dd-467b-830e-47b68c1ea4e0 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:05:14.903751 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f9c15fc-47dd-467b-830e-47b68c1ea4e0 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:05:14.904114 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f9c15fc-47dd-467b-830e-47b68c1ea4e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:05:14.925642 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-5f9c15fc-47dd-467b-830e-47b68c1ea4e0 None None] MFA Rules not processed for user `f4b0e9187ce74f7888e9e7094be5c824`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:05:14.928531 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-5f9c15fc-47dd-467b-830e-47b68c1ea4e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:05:14.998510 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 984/1974] 10.222.0.36 () {58 vars in 1009 bytes} [Thu Oct 20 09:05:14 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 20 09:05:15.069319 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-366ffe1f-771c-4352-9289-3fed4c5ef29c service glance] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:05:15.070398 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-366ffe1f-771c-4352-9289-3fed4c5ef29c service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:05:15.154217 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:05:15.160888 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-366ffe1f-771c-4352-9289-3fed4c5ef29c service glance] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:05:15.167078 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5d2df81-03f4-4a65-865e-e85629299fb3 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4d558918cd8b4ebea4fe8765297304db', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:05:15.169722 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5d2df81-03f4-4a65-865e-e85629299fb3 service swift] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:05:15.169722 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5d2df81-03f4-4a65-865e-e85629299fb3 service swift] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:05:15.169722 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5d2df81-03f4-4a65-865e-e85629299fb3 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:05:15.180554 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5d2df81-03f4-4a65-865e-e85629299fb3 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:05:15.182414 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5d2df81-03f4-4a65-865e-e85629299fb3 service swift] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:05:15.192334 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 991/1975] 10.222.0.36 () {60 vars in 1384 bytes} [Thu Oct 20 09:05:15 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 124 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 20 09:05:24.251579 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45189baf-47d1-4e45-b885-3d21560e7333 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:05:24.252089 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45189baf-47d1-4e45-b885-3d21560e7333 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:05:24.252197 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45189baf-47d1-4e45-b885-3d21560e7333 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:05:24.272795 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-45189baf-47d1-4e45-b885-3d21560e7333 None None] MFA Rules not processed for user `14ddd15ab87a4d0fbea1d6dfa910f8ce`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:05:24.274079 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-45189baf-47d1-4e45-b885-3d21560e7333 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:05:24.287314 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 985/1976] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:05:24 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:05:25.188820 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=5 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:05:27.451647 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3f6a38-4bc9-406d-897c-d0729a5a4ed1 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:05:27.451647 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3f6a38-4bc9-406d-897c-d0729a5a4ed1 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:05:27.451647 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3f6a38-4bc9-406d-897c-d0729a5a4ed1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:05:27.472331 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-ac3f6a38-4bc9-406d-897c-d0729a5a4ed1 None None] MFA Rules not processed for user `f4b0e9187ce74f7888e9e7094be5c824`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:05:27.473781 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-ac3f6a38-4bc9-406d-897c-d0729a5a4ed1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:05:27.481041 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 992/1977] 10.222.0.36 () {58 vars in 1009 bytes} [Thu Oct 20 09:05:27 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:05:27.494138 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45189baf-47d1-4e45-b885-3d21560e7333 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:05:27.506404 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45189baf-47d1-4e45-b885-3d21560e7333 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:05:27.511831 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca12c26f-7816-48d6-8405-155ca65975a0 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4d558918cd8b4ebea4fe8765297304db', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:05:27.512441 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca12c26f-7816-48d6-8405-155ca65975a0 service swift] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:05:27.512713 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca12c26f-7816-48d6-8405-155ca65975a0 service swift] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:05:27.512974 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca12c26f-7816-48d6-8405-155ca65975a0 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:05:27.519301 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:05:27.521792 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca12c26f-7816-48d6-8405-155ca65975a0 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:05:27.524050 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca12c26f-7816-48d6-8405-155ca65975a0 service swift] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:05:27.534213 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 986/1978] 10.222.0.36 () {60 vars in 1384 bytes} [Thu Oct 20 09:05:27 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 20 09:05:28.701035 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac3f6a38-4bc9-406d-897c-d0729a5a4ed1 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:05:28.757546 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac3f6a38-4bc9-406d-897c-d0729a5a4ed1 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:05:28.761220 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18db1257-e055-46d2-87cc-fcbf23619b12 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:05:28.761832 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18db1257-e055-46d2-87cc-fcbf23619b12 service neutron] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:05:28.762126 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18db1257-e055-46d2-87cc-fcbf23619b12 service neutron] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:05:28.762381 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18db1257-e055-46d2-87cc-fcbf23619b12 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:05:28.767312 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-18db1257-e055-46d2-87cc-fcbf23619b12 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:05:28.844002 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18db1257-e055-46d2-87cc-fcbf23619b12 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:05:28.845644 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18db1257-e055-46d2-87cc-fcbf23619b12 service neutron] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:05:28.857458 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 993/1979] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:05:28 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 157 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:05:30.055533 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca12c26f-7816-48d6-8405-155ca65975a0 service swift] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:05:30.057051 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-ca12c26f-7816-48d6-8405-155ca65975a0 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:05:30.068828 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca12c26f-7816-48d6-8405-155ca65975a0 service swift] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:05:30.072759 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf1376a0-bd5a-43c0-bedc-c24502aca8f1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:05:30.073401 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf1376a0-bd5a-43c0-bedc-c24502aca8f1 service neutron] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:05:30.073679 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf1376a0-bd5a-43c0-bedc-c24502aca8f1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:05:30.073948 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf1376a0-bd5a-43c0-bedc-c24502aca8f1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:05:30.075715 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-bf1376a0-bd5a-43c0-bedc-c24502aca8f1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:05:30.093431 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf1376a0-bd5a-43c0-bedc-c24502aca8f1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:05:30.095420 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf1376a0-bd5a-43c0-bedc-c24502aca8f1 service neutron] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:05:30.108821 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 987/1980] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:05:30 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 54 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:05:38.851903 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=57 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:05:40.102149 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=8 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:05:41.385594 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18db1257-e055-46d2-87cc-fcbf23619b12 service neutron] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:05:41.387720 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-18db1257-e055-46d2-87cc-fcbf23619b12 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:05:41.402156 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18db1257-e055-46d2-87cc-fcbf23619b12 service neutron] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:05:41.407857 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-399dc83b-7ff6-44e5-9c72-abbeef4e7535 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a94e868115ec49918eda10724263930c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:05:41.408480 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-399dc83b-7ff6-44e5-9c72-abbeef4e7535 service nova] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:05:41.408753 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-399dc83b-7ff6-44e5-9c72-abbeef4e7535 service nova] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:05:41.409021 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-399dc83b-7ff6-44e5-9c72-abbeef4e7535 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:05:41.410810 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-399dc83b-7ff6-44e5-9c72-abbeef4e7535 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:05:41.473022 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-399dc83b-7ff6-44e5-9c72-abbeef4e7535 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:05:41.474817 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-399dc83b-7ff6-44e5-9c72-abbeef4e7535 service nova] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:05:41.484379 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 994/1981] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:05:41 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:05:45.891449 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf1376a0-bd5a-43c0-bedc-c24502aca8f1 service neutron] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:05:45.903509 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf1376a0-bd5a-43c0-bedc-c24502aca8f1 service neutron] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:05:45.909089 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-723c166e-f319-48c2-8eca-c8c8ca58d36f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a94e868115ec49918eda10724263930c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:05:45.909794 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723c166e-f319-48c2-8eca-c8c8ca58d36f service nova] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:05:45.910258 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723c166e-f319-48c2-8eca-c8c8ca58d36f service nova] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:05:45.910657 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723c166e-f319-48c2-8eca-c8c8ca58d36f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:05:45.912515 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-723c166e-f319-48c2-8eca-c8c8ca58d36f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:05:45.987513 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-723c166e-f319-48c2-8eca-c8c8ca58d36f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:05:45.989470 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-723c166e-f319-48c2-8eca-c8c8ca58d36f service nova] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:05:45.998762 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 988/1982] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:05:45 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:05:51.480535 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=28 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:05:55.995015 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=28 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:05:56.086275 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ebd2cd2-07f9-43c1-8a54-ca0a782f8cd5 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:05:56.086797 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ebd2cd2-07f9-43c1-8a54-ca0a782f8cd5 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:05:56.086896 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ebd2cd2-07f9-43c1-8a54-ca0a782f8cd5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:05:56.106324 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-6ebd2cd2-07f9-43c1-8a54-ca0a782f8cd5 None None] MFA Rules not processed for user `8002ab26f6d64822aaa65aa730781b76`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:05:56.107544 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-6ebd2cd2-07f9-43c1-8a54-ca0a782f8cd5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:05:56.114051 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 995/1983] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:05:56 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:05:56.417286 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-723c166e-f319-48c2-8eca-c8c8ca58d36f service nova] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:05:56.426651 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-723c166e-f319-48c2-8eca-c8c8ca58d36f service nova] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:05:56.430223 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-200b5f6f-0e36-4f1f-b9c6-c98fec72bde7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:05:56.430847 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-200b5f6f-0e36-4f1f-b9c6-c98fec72bde7 service neutron] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:05:56.431120 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-200b5f6f-0e36-4f1f-b9c6-c98fec72bde7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:05:56.431386 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-200b5f6f-0e36-4f1f-b9c6-c98fec72bde7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:05:56.433648 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-200b5f6f-0e36-4f1f-b9c6-c98fec72bde7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:05:56.442907 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-200b5f6f-0e36-4f1f-b9c6-c98fec72bde7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:05:56.444296 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-200b5f6f-0e36-4f1f-b9c6-c98fec72bde7 service neutron] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:05:56.452476 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 989/1984] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:05:56 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:05:56.766000 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ebd2cd2-07f9-43c1-8a54-ca0a782f8cd5 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:05:56.767478 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-6ebd2cd2-07f9-43c1-8a54-ca0a782f8cd5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:05:56.861610 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ebd2cd2-07f9-43c1-8a54-ca0a782f8cd5 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:05:56.866681 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e8ca72d-6d1c-4e41-aebe-b37627214e7c service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ad03ab76b9c44b1cb369475afa25eddd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:05:56.867290 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e8ca72d-6d1c-4e41-aebe-b37627214e7c service placement] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:05:56.867566 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e8ca72d-6d1c-4e41-aebe-b37627214e7c service placement] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:05:56.867835 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e8ca72d-6d1c-4e41-aebe-b37627214e7c service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:05:56.869527 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-1e8ca72d-6d1c-4e41-aebe-b37627214e7c service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:05:56.881116 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e8ca72d-6d1c-4e41-aebe-b37627214e7c service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:05:56.882650 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e8ca72d-6d1c-4e41-aebe-b37627214e7c service placement] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:05:56.892867 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 996/1985] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:05:56 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 127 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:06:00.325672 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-200b5f6f-0e36-4f1f-b9c6-c98fec72bde7 service neutron] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:06:00.334027 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-200b5f6f-0e36-4f1f-b9c6-c98fec72bde7 service neutron] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:06:00.337696 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be527eb0-23a7-47a9-b018-3c6f6c86291d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:06:00.338264 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be527eb0-23a7-47a9-b018-3c6f6c86291d service neutron] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:00.338568 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be527eb0-23a7-47a9-b018-3c6f6c86291d service neutron] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:00.339403 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be527eb0-23a7-47a9-b018-3c6f6c86291d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:00.349037 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be527eb0-23a7-47a9-b018-3c6f6c86291d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:06:00.350448 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be527eb0-23a7-47a9-b018-3c6f6c86291d service neutron] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:06:00.359791 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 990/1986] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:06:00 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 20 09:06:01.391118 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e8ca72d-6d1c-4e41-aebe-b37627214e7c service placement] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:06:01.400967 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e8ca72d-6d1c-4e41-aebe-b37627214e7c service placement] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:06:01.405650 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4b87419-a249-47ea-b3f6-899bdf00a15e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:06:01.406251 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b87419-a249-47ea-b3f6-899bdf00a15e service neutron] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:01.406568 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b87419-a249-47ea-b3f6-899bdf00a15e service neutron] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:01.406845 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b87419-a249-47ea-b3f6-899bdf00a15e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:01.408580 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-f4b87419-a249-47ea-b3f6-899bdf00a15e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:06:01.473739 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4b87419-a249-47ea-b3f6-899bdf00a15e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:06:01.475277 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4b87419-a249-47ea-b3f6-899bdf00a15e service neutron] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:06:01.486163 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 997/1987] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:06:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:06:02.501548 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be527eb0-23a7-47a9-b018-3c6f6c86291d service neutron] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:06:02.512880 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be527eb0-23a7-47a9-b018-3c6f6c86291d service neutron] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:06:02.524350 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b455c9e1-d2d2-4273-a769-5ec7ecd59684 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:06:02.524972 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b455c9e1-d2d2-4273-a769-5ec7ecd59684 service neutron] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:02.525250 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b455c9e1-d2d2-4273-a769-5ec7ecd59684 service neutron] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:02.525523 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b455c9e1-d2d2-4273-a769-5ec7ecd59684 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:02.527586 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-b455c9e1-d2d2-4273-a769-5ec7ecd59684 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:06:02.540257 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b455c9e1-d2d2-4273-a769-5ec7ecd59684 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:06:02.542991 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b455c9e1-d2d2-4273-a769-5ec7ecd59684 service neutron] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:06:02.558041 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 991/1988] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:06:02 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 57 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:06:05.228079 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4b87419-a249-47ea-b3f6-899bdf00a15e service neutron] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:06:05.244341 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4b87419-a249-47ea-b3f6-899bdf00a15e service neutron] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:06:05.250383 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fea63b14-8007-4a4b-a6ca-d929dd2669df service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a94e868115ec49918eda10724263930c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:06:05.251062 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea63b14-8007-4a4b-a6ca-d929dd2669df service nova] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:05.251335 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea63b14-8007-4a4b-a6ca-d929dd2669df service nova] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:05.251602 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea63b14-8007-4a4b-a6ca-d929dd2669df service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:05.253371 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-fea63b14-8007-4a4b-a6ca-d929dd2669df service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:06:05.266161 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fea63b14-8007-4a4b-a6ca-d929dd2669df service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:06:05.267717 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fea63b14-8007-4a4b-a6ca-d929dd2669df service nova] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:06:05.280505 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 998/1989] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:06:05 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 53 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:06:06.667973 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b455c9e1-d2d2-4273-a769-5ec7ecd59684 service neutron] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:06:06.669760 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-b455c9e1-d2d2-4273-a769-5ec7ecd59684 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:06:06.695134 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b455c9e1-d2d2-4273-a769-5ec7ecd59684 service neutron] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:06:06.700672 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aca35dc0-1ae3-454c-ab2c-ff053a12e991 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:06:06.701429 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca35dc0-1ae3-454c-ab2c-ff053a12e991 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:06.701789 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca35dc0-1ae3-454c-ab2c-ff053a12e991 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:06.702068 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca35dc0-1ae3-454c-ab2c-ff053a12e991 admin admin] PATH_INFO: `/v3/users/7f932695a5994c5790d14c210ea65326` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:06.703358 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aca35dc0-1ae3-454c-ab2c-ff053a12e991 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7f932695a5994c5790d14c210ea65326)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:06:06.705428 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aca35dc0-1ae3-454c-ab2c-ff053a12e991 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:06:06.764157 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-aca35dc0-1ae3-454c-ab2c-ff053a12e991 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7f932695a5994c5790d14c210ea65326'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:06:06.773631 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-aca35dc0-1ae3-454c-ab2c-ff053a12e991 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7f932695a5994c5790d14c210ea65326'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:06:06.786710 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-aca35dc0-1ae3-454c-ab2c-ff053a12e991 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7f932695a5994c5790d14c210ea65326'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:06:06.794413 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-aca35dc0-1ae3-454c-ab2c-ff053a12e991 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7f932695a5994c5790d14c210ea65326'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:06:06.794911 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-aca35dc0-1ae3-454c-ab2c-ff053a12e991 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7f932695a5994c5790d14c210ea65326'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:06:06.800762 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 992/1990] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:06:06 2022] DELETE /identity/v3/users/7f932695a5994c5790d14c210ea65326 => generated 0 bytes in 133 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:06:06.974403 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fea63b14-8007-4a4b-a6ca-d929dd2669df service nova] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:06:06.988538 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fea63b14-8007-4a4b-a6ca-d929dd2669df service nova] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:06:07.049085 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fbba277-64a6-46c4-a27f-fb78a6c2bd74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:06:07.049637 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fbba277-64a6-46c4-a27f-fb78a6c2bd74 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:07.049914 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fbba277-64a6-46c4-a27f-fb78a6c2bd74 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:07.050315 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fbba277-64a6-46c4-a27f-fb78a6c2bd74 admin admin] PATH_INFO: `/v3/projects/d24a1fc0e9344af181d49cfc58f0c88b` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:07.051460 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fbba277-64a6-46c4-a27f-fb78a6c2bd74 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d24a1fc0e9344af181d49cfc58f0c88b)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:06:07.053403 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fbba277-64a6-46c4-a27f-fb78a6c2bd74 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:06:07.059681 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:06:07.101692 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-3fbba277-64a6-46c4-a27f-fb78a6c2bd74 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd24a1fc0e9344af181d49cfc58f0c88b', 'request_id': 'req-3fbba277-64a6-46c4-a27f-fb78a6c2bd74'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:06:07.107875 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-3fbba277-64a6-46c4-a27f-fb78a6c2bd74 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd24a1fc0e9344af181d49cfc58f0c88b', 'request_id': 'req-3fbba277-64a6-46c4-a27f-fb78a6c2bd74'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:06:07.113389 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-3fbba277-64a6-46c4-a27f-fb78a6c2bd74 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd24a1fc0e9344af181d49cfc58f0c88b', 'request_id': 'req-3fbba277-64a6-46c4-a27f-fb78a6c2bd74'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:06:07.125525 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-3fbba277-64a6-46c4-a27f-fb78a6c2bd74 admin admin] The token cache is being invalidate because project d24a1fc0e9344af181d49cfc58f0c88b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81561) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:06:07.125812 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-3fbba277-64a6-46c4-a27f-fb78a6c2bd74 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:06:07.126518 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-3fbba277-64a6-46c4-a27f-fb78a6c2bd74 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:06:07.128020 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 999/1991] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:06:06 2022] DELETE /identity/v3/projects/d24a1fc0e9344af181d49cfc58f0c88b => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:06:07.132924 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d757c120-ee92-468b-a396-da6b68a53c54 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:07.133209 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d757c120-ee92-468b-a396-da6b68a53c54 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:07.133476 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d757c120-ee92-468b-a396-da6b68a53c54 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:07.153616 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-d757c120-ee92-468b-a396-da6b68a53c54 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:06:07.154824 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-d757c120-ee92-468b-a396-da6b68a53c54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:06:07.213148 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 993/1992] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:06:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:06:07.221415 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce588c4-8867-4b47-9aca-cbf25374fc31 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:07.221769 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce588c4-8867-4b47-9aca-cbf25374fc31 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:07.222040 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce588c4-8867-4b47-9aca-cbf25374fc31 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:07.245841 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-2ce588c4-8867-4b47-9aca-cbf25374fc31 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:06:07.247082 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-2ce588c4-8867-4b47-9aca-cbf25374fc31 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:06:07.255000 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1000/1993] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:06:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:06:07.259722 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d757c120-ee92-468b-a396-da6b68a53c54 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:06:07.269720 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d757c120-ee92-468b-a396-da6b68a53c54 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:06:07.274295 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c407e00-7b44-43c4-98a7-7e5bc22afd58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:06:07.274931 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c407e00-7b44-43c4-98a7-7e5bc22afd58 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:07.275212 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c407e00-7b44-43c4-98a7-7e5bc22afd58 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:07.275483 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c407e00-7b44-43c4-98a7-7e5bc22afd58 admin admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:07.276263 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c407e00-7b44-43c4-98a7-7e5bc22afd58 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 09:06:07.276564 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c407e00-7b44-43c4-98a7-7e5bc22afd58 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:06:07.278175 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c407e00-7b44-43c4-98a7-7e5bc22afd58 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:06:07.289981 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 994/1994] 10.222.0.36 () {58 vars in 1153 bytes} [Thu Oct 20 09:06:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 09:06:07.294853 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce588c4-8867-4b47-9aca-cbf25374fc31 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:06:07.304734 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce588c4-8867-4b47-9aca-cbf25374fc31 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:06:07.309974 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99a7ebdd-53ea-4bff-92d7-8ba61df2d48c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:06:07.310605 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99a7ebdd-53ea-4bff-92d7-8ba61df2d48c admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:07.310883 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99a7ebdd-53ea-4bff-92d7-8ba61df2d48c admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:07.311149 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99a7ebdd-53ea-4bff-92d7-8ba61df2d48c admin admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:07.311770 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99a7ebdd-53ea-4bff-92d7-8ba61df2d48c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:06:07.313684 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99a7ebdd-53ea-4bff-92d7-8ba61df2d48c admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:06:07.344049 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1001/1995] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:06:07 2022] POST /identity/v3/projects => generated 373 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:06:07.345241 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c407e00-7b44-43c4-98a7-7e5bc22afd58 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:06:07.356275 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c407e00-7b44-43c4-98a7-7e5bc22afd58 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:06:07.374449 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:06:07.419117 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b6d8a91-ab1d-463d-b05c-4df7d4d1abac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:06:07.419716 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b6d8a91-ab1d-463d-b05c-4df7d4d1abac admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:07.419988 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b6d8a91-ab1d-463d-b05c-4df7d4d1abac admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:07.420254 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b6d8a91-ab1d-463d-b05c-4df7d4d1abac admin admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:07.420867 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b6d8a91-ab1d-463d-b05c-4df7d4d1abac admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:06:07.422789 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b6d8a91-ab1d-463d-b05c-4df7d4d1abac admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:06:07.442995 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 995/1996] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:06:07 2022] POST /identity/v3/users => generated 287 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:06:07.447100 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99a7ebdd-53ea-4bff-92d7-8ba61df2d48c admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:06:07.457263 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99a7ebdd-53ea-4bff-92d7-8ba61df2d48c admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:06:07.461854 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf5c3597-8886-41f1-9d78-4cf6f817ad59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:06:07.462438 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf5c3597-8886-41f1-9d78-4cf6f817ad59 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:07.462741 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf5c3597-8886-41f1-9d78-4cf6f817ad59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:07.463007 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf5c3597-8886-41f1-9d78-4cf6f817ad59 admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:07.464185 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf5c3597-8886-41f1-9d78-4cf6f817ad59 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:06:07.466026 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf5c3597-8886-41f1-9d78-4cf6f817ad59 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:06:07.475802 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1002/1997] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:06:07 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:06:07.480086 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b6d8a91-ab1d-463d-b05c-4df7d4d1abac admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:06:07.489569 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b6d8a91-ab1d-463d-b05c-4df7d4d1abac admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:06:07.494134 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f27beea2-48f8-4f2d-8eac-4fd92eba47af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:06:07.495346 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f27beea2-48f8-4f2d-8eac-4fd92eba47af admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:07.495620 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f27beea2-48f8-4f2d-8eac-4fd92eba47af admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:07.495883 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f27beea2-48f8-4f2d-8eac-4fd92eba47af admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:07.496517 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f27beea2-48f8-4f2d-8eac-4fd92eba47af admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:06:07.498091 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f27beea2-48f8-4f2d-8eac-4fd92eba47af admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:06:07.508762 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 996/1998] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:06:07 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:06:07.513502 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf5c3597-8886-41f1-9d78-4cf6f817ad59 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:06:07.523809 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf5c3597-8886-41f1-9d78-4cf6f817ad59 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:06:07.528550 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72d7f3bd-4ce0-489d-aee3-e80dab7c83c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:06:07.529191 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d7f3bd-4ce0-489d-aee3-e80dab7c83c9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:07.529481 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d7f3bd-4ce0-489d-aee3-e80dab7c83c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:07.529739 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d7f3bd-4ce0-489d-aee3-e80dab7c83c9 admin admin] PATH_INFO: `/v3/projects/9c6705155a634a18a0fda600155f6457/users/07a6576d403b4cbb8e78c7efd00153a3/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:07.544327 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72d7f3bd-4ce0-489d-aee3-e80dab7c83c9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9c6705155a634a18a0fda600155f6457, user_id=07a6576d403b4cbb8e78c7efd00153a3, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:06:07.547033 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72d7f3bd-4ce0-489d-aee3-e80dab7c83c9 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:06:07.560659 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1003/1999] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:06:07 2022] PUT /identity/v3/projects/9c6705155a634a18a0fda600155f6457/users/07a6576d403b4cbb8e78c7efd00153a3/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:06:07.571499 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611fac90-e154-48e8-bc0e-f87245cb1677 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:07.571910 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611fac90-e154-48e8-bc0e-f87245cb1677 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:07.572302 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611fac90-e154-48e8-bc0e-f87245cb1677 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:07.601671 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-611fac90-e154-48e8-bc0e-f87245cb1677 None None] MFA Rules not processed for user `07a6576d403b4cbb8e78c7efd00153a3`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:06:07.603015 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-611fac90-e154-48e8-bc0e-f87245cb1677 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:06:07.668665 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 997/2000] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:06:07 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:06:07.673105 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72d7f3bd-4ce0-489d-aee3-e80dab7c83c9 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:06:07.747804 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72d7f3bd-4ce0-489d-aee3-e80dab7c83c9 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:06:07.752758 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-019cff1b-47c3-48e7-bace-584c1f7e5494 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:06:07.753360 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-019cff1b-47c3-48e7-bace-584c1f7e5494 admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:07.753643 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-019cff1b-47c3-48e7-bace-584c1f7e5494 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:07.753910 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-019cff1b-47c3-48e7-bace-584c1f7e5494 admin admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:07.754997 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-019cff1b-47c3-48e7-bace-584c1f7e5494 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:06:07.756666 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-019cff1b-47c3-48e7-bace-584c1f7e5494 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:06:07.787725 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1004/2001] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:06:07 2022] POST /identity/v3/projects => generated 371 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:06:07.792309 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-611fac90-e154-48e8-bc0e-f87245cb1677 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:06:07.803962 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-611fac90-e154-48e8-bc0e-f87245cb1677 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:06:07.862484 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8cabfea-f4d7-4c7a-a77a-52c3f1fe306a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:06:07.863097 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8cabfea-f4d7-4c7a-a77a-52c3f1fe306a admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:07.863369 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8cabfea-f4d7-4c7a-a77a-52c3f1fe306a admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:07.863633 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8cabfea-f4d7-4c7a-a77a-52c3f1fe306a admin admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:07.864259 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8cabfea-f4d7-4c7a-a77a-52c3f1fe306a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:06:07.866190 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8cabfea-f4d7-4c7a-a77a-52c3f1fe306a admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:06:07.888557 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 998/2002] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:06:07 2022] POST /identity/v3/users => generated 292 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 20 09:06:07.892792 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-019cff1b-47c3-48e7-bace-584c1f7e5494 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:06:07.903288 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-019cff1b-47c3-48e7-bace-584c1f7e5494 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:06:07.907948 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6ce1e34-66b8-46c1-8d7d-00a1b5b0b545 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:06:07.908541 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ce1e34-66b8-46c1-8d7d-00a1b5b0b545 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:07.908812 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ce1e34-66b8-46c1-8d7d-00a1b5b0b545 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:07.909195 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ce1e34-66b8-46c1-8d7d-00a1b5b0b545 admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:07.909831 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6ce1e34-66b8-46c1-8d7d-00a1b5b0b545 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:06:07.911489 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6ce1e34-66b8-46c1-8d7d-00a1b5b0b545 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:06:07.926136 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1005/2003] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:06:07 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:06:07.927472 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8cabfea-f4d7-4c7a-a77a-52c3f1fe306a admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:06:07.942409 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8cabfea-f4d7-4c7a-a77a-52c3f1fe306a admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:06:07.947274 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc9a29bc-5219-4595-822d-09fb54323aaa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:06:07.947963 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9a29bc-5219-4595-822d-09fb54323aaa admin admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:07.948238 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9a29bc-5219-4595-822d-09fb54323aaa admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:07.948507 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9a29bc-5219-4595-822d-09fb54323aaa admin admin] PATH_INFO: `/v3/projects/1a8ca1fdf8654505b50fc13ce94c598b/users/ce32c50e27754b65984f88ca1773f0c7/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:07.964141 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc9a29bc-5219-4595-822d-09fb54323aaa admin admin] RBAC: Authorizing `identity:create_grant(project_id=1a8ca1fdf8654505b50fc13ce94c598b, user_id=ce32c50e27754b65984f88ca1773f0c7, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:06:07.967177 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc9a29bc-5219-4595-822d-09fb54323aaa admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:06:07.970972 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:06:07.989259 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 999/2004] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:06:07 2022] PUT /identity/v3/projects/1a8ca1fdf8654505b50fc13ce94c598b/users/ce32c50e27754b65984f88ca1773f0c7/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 62 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:06:07.993724 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6ce1e34-66b8-46c1-8d7d-00a1b5b0b545 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:06:08.001419 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:06:08.006464 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6ce1e34-66b8-46c1-8d7d-00a1b5b0b545 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:06:08.065838 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a71c46b5-6a46-4861-a587-6335d982f845 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:06:08.066441 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a71c46b5-6a46-4861-a587-6335d982f845 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:08.066751 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a71c46b5-6a46-4861-a587-6335d982f845 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:08.067022 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a71c46b5-6a46-4861-a587-6335d982f845 admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:08.067654 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a71c46b5-6a46-4861-a587-6335d982f845 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:06:08.069266 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a71c46b5-6a46-4861-a587-6335d982f845 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:06:08.079625 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1006/2005] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:06:07 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:06:08.083799 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc9a29bc-5219-4595-822d-09fb54323aaa admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:06:08.094312 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc9a29bc-5219-4595-822d-09fb54323aaa admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:06:08.099300 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7592925-c3b4-40c0-a821-8e4d1526407b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:06:08.100170 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7592925-c3b4-40c0-a821-8e4d1526407b admin admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:08.100625 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7592925-c3b4-40c0-a821-8e4d1526407b admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:08.101066 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7592925-c3b4-40c0-a821-8e4d1526407b admin admin] PATH_INFO: `/v3/projects/1a8ca1fdf8654505b50fc13ce94c598b/users/ce32c50e27754b65984f88ca1773f0c7/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:08.103237 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7592925-c3b4-40c0-a821-8e4d1526407b admin admin] RBAC: Authorizing `identity:create_grant(project_id=1a8ca1fdf8654505b50fc13ce94c598b, user_id=ce32c50e27754b65984f88ca1773f0c7, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:06:08.106539 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7592925-c3b4-40c0-a821-8e4d1526407b admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:06:08.122914 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1000/2006] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:06:08 2022] PUT /identity/v3/projects/1a8ca1fdf8654505b50fc13ce94c598b/users/ce32c50e27754b65984f88ca1773f0c7/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 39 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:06:08.132699 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4147fde0-a030-4b28-997e-3329cac5d7fc None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:08.132988 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4147fde0-a030-4b28-997e-3329cac5d7fc None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:08.133259 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4147fde0-a030-4b28-997e-3329cac5d7fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:08.166335 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-4147fde0-a030-4b28-997e-3329cac5d7fc None None] MFA Rules not processed for user `ce32c50e27754b65984f88ca1773f0c7`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:06:08.168988 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-4147fde0-a030-4b28-997e-3329cac5d7fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:06:08.246162 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1007/2007] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:06:08 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:06:08.257746 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7592925-c3b4-40c0-a821-8e4d1526407b admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:06:08.259476 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-b7592925-c3b4-40c0-a821-8e4d1526407b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:06:08.321139 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7592925-c3b4-40c0-a821-8e4d1526407b admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:06:08.325240 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de0c1d81-466e-4706-9b07-6a32b8e72a8f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:06:08.325954 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0c1d81-466e-4706-9b07-6a32b8e72a8f service cinder] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:08.326324 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0c1d81-466e-4706-9b07-6a32b8e72a8f service cinder] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:08.326727 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0c1d81-466e-4706-9b07-6a32b8e72a8f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:08.336847 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de0c1d81-466e-4706-9b07-6a32b8e72a8f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:06:08.337983 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de0c1d81-466e-4706-9b07-6a32b8e72a8f service cinder] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:06:08.348757 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1001/2008] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:06:08 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:06:08.438569 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4147fde0-a030-4b28-997e-3329cac5d7fc None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:06:08.450729 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4147fde0-a030-4b28-997e-3329cac5d7fc None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:06:08.457118 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfe25262-8dfd-4b78-9d1b-707d77899c58 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:06:08.457743 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe25262-8dfd-4b78-9d1b-707d77899c58 service cinder] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:08.458018 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe25262-8dfd-4b78-9d1b-707d77899c58 service cinder] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:08.458287 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe25262-8dfd-4b78-9d1b-707d77899c58 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:08.523269 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfe25262-8dfd-4b78-9d1b-707d77899c58 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:06:08.523719 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfe25262-8dfd-4b78-9d1b-707d77899c58 service cinder] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:06:08.533870 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1008/2009] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:06:08 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:06:09.224470 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de0c1d81-466e-4706-9b07-6a32b8e72a8f service cinder] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:06:09.226058 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-de0c1d81-466e-4706-9b07-6a32b8e72a8f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:06:09.301959 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de0c1d81-466e-4706-9b07-6a32b8e72a8f service cinder] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:06:09.308050 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61f837cc-a733-40f0-8f54-acb0717a955e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a94e868115ec49918eda10724263930c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:06:09.308700 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f837cc-a733-40f0-8f54-acb0717a955e service nova] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:09.308973 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f837cc-a733-40f0-8f54-acb0717a955e service nova] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:09.309240 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f837cc-a733-40f0-8f54-acb0717a955e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:09.311420 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-61f837cc-a733-40f0-8f54-acb0717a955e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:06:09.372383 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61f837cc-a733-40f0-8f54-acb0717a955e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:06:09.374011 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61f837cc-a733-40f0-8f54-acb0717a955e service nova] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:06:09.383802 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1002/2010] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:06:09 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 160 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 20 09:06:12.882317 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9564820c-3d56-46a7-871b-24fb97303025 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:12.882904 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9564820c-3d56-46a7-871b-24fb97303025 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:12.882904 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9564820c-3d56-46a7-871b-24fb97303025 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:12.902228 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-9564820c-3d56-46a7-871b-24fb97303025 None None] MFA Rules not processed for user `be1f82d7a0a84fbebdd0dffef9f098ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:06:12.903446 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-9564820c-3d56-46a7-871b-24fb97303025 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:06:12.922351 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:06:12.961491 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1009/2011] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:06:12 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:06:13.262241 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61f837cc-a733-40f0-8f54-acb0717a955e service nova] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:06:13.263686 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-61f837cc-a733-40f0-8f54-acb0717a955e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:06:13.284750 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:INSERT=2,keystone:SELECT=98 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:06:13.320691 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61f837cc-a733-40f0-8f54-acb0717a955e service nova] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:06:13.324340 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba919df1-b423-459e-a69e-23e75b6b3312 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:06:13.324937 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba919df1-b423-459e-a69e-23e75b6b3312 service neutron] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:13.325215 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba919df1-b423-459e-a69e-23e75b6b3312 service neutron] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:13.325485 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba919df1-b423-459e-a69e-23e75b6b3312 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:13.327238 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-ba919df1-b423-459e-a69e-23e75b6b3312 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:06:13.336422 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba919df1-b423-459e-a69e-23e75b6b3312 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:06:13.337845 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba919df1-b423-459e-a69e-23e75b6b3312 service neutron] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:06:13.345969 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1003/2012] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:06:13 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:06:16.283575 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9564820c-3d56-46a7-871b-24fb97303025 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:06:16.285168 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-9564820c-3d56-46a7-871b-24fb97303025 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:06:16.301134 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9564820c-3d56-46a7-871b-24fb97303025 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:06:16.306530 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5d1bb32-b24c-4a5f-841b-7e872aa9aefe service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a94e868115ec49918eda10724263930c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:06:16.307197 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d1bb32-b24c-4a5f-841b-7e872aa9aefe service nova] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:16.307567 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d1bb32-b24c-4a5f-841b-7e872aa9aefe service nova] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:16.307956 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d1bb32-b24c-4a5f-841b-7e872aa9aefe service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:16.310236 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-d5d1bb32-b24c-4a5f-841b-7e872aa9aefe service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:06:16.322113 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5d1bb32-b24c-4a5f-841b-7e872aa9aefe service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:06:16.323945 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5d1bb32-b24c-4a5f-841b-7e872aa9aefe service nova] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:06:16.334804 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1010/2013] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:06:16 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:06:22.796412 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba919df1-b423-459e-a69e-23e75b6b3312 service neutron] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:06:22.813182 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba919df1-b423-459e-a69e-23e75b6b3312 service neutron] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:06:22.820574 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8c7c75f-08e1-4e8c-a8de-fe077bd5d76f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a94e868115ec49918eda10724263930c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:06:22.821658 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c7c75f-08e1-4e8c-a8de-fe077bd5d76f service nova] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:22.822053 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c7c75f-08e1-4e8c-a8de-fe077bd5d76f service nova] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:22.822562 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c7c75f-08e1-4e8c-a8de-fe077bd5d76f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:22.824896 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-b8c7c75f-08e1-4e8c-a8de-fe077bd5d76f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:06:22.839499 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8c7c75f-08e1-4e8c-a8de-fe077bd5d76f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:06:22.842246 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8c7c75f-08e1-4e8c-a8de-fe077bd5d76f service nova] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:06:22.854506 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1004/2014] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:06:22 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 59 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:06:26.330152 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=20 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:06:32.848924 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=23 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:06:40.140622 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1ef22a-6acf-4056-a961-4ff9631bff11 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:40.141111 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1ef22a-6acf-4056-a961-4ff9631bff11 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:40.141213 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1ef22a-6acf-4056-a961-4ff9631bff11 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:40.162899 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-8c1ef22a-6acf-4056-a961-4ff9631bff11 None None] MFA Rules not processed for user `65ca2eae6db24bb094afaacdfb5c1980`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:06:40.164143 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-8c1ef22a-6acf-4056-a961-4ff9631bff11 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:06:40.220639 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1011/2015] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:06:40 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:06:40.440031 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f5a44b-cac0-4ddd-a6a1-ed87a143ca95 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:40.440537 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f5a44b-cac0-4ddd-a6a1-ed87a143ca95 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:40.440643 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f5a44b-cac0-4ddd-a6a1-ed87a143ca95 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:40.461252 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-63f5a44b-cac0-4ddd-a6a1-ed87a143ca95 None None] MFA Rules not processed for user `f4b0e9187ce74f7888e9e7094be5c824`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:06:40.462463 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-63f5a44b-cac0-4ddd-a6a1-ed87a143ca95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:06:40.518662 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1005/2016] 10.222.0.36 () {58 vars in 1009 bytes} [Thu Oct 20 09:06:40 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:06:40.562196 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1ef22a-6acf-4056-a961-4ff9631bff11 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:06:40.563629 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-8c1ef22a-6acf-4056-a961-4ff9631bff11 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:06:40.637295 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1ef22a-6acf-4056-a961-4ff9631bff11 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:06:40.642963 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f7e231d-2ed5-4b89-ab24-8ddcb87d2e5d service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4d558918cd8b4ebea4fe8765297304db', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:06:40.643852 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7e231d-2ed5-4b89-ab24-8ddcb87d2e5d service swift] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:40.644132 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7e231d-2ed5-4b89-ab24-8ddcb87d2e5d service swift] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:40.644404 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7e231d-2ed5-4b89-ab24-8ddcb87d2e5d service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:40.653001 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f7e231d-2ed5-4b89-ab24-8ddcb87d2e5d service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:06:40.655159 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f7e231d-2ed5-4b89-ab24-8ddcb87d2e5d service swift] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:06:40.664496 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1012/2017] 10.222.0.36 () {60 vars in 1384 bytes} [Thu Oct 20 09:06:40 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 20 09:06:50.512709 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=25 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:06:50.572772 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63f5a44b-cac0-4ddd-a6a1-ed87a143ca95 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:06:50.642217 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63f5a44b-cac0-4ddd-a6a1-ed87a143ca95 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:06:50.647175 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfe26c5a-b13d-4b1d-b8f2-5551d6de55fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:06:50.647946 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe26c5a-b13d-4b1d-b8f2-5551d6de55fc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:50.648362 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe26c5a-b13d-4b1d-b8f2-5551d6de55fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:50.648722 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe26c5a-b13d-4b1d-b8f2-5551d6de55fc admin admin] PATH_INFO: `/v3/users/07a6576d403b4cbb8e78c7efd00153a3` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:50.649991 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfe26c5a-b13d-4b1d-b8f2-5551d6de55fc admin admin] RBAC: Authorizing `identity:delete_user(user_id=07a6576d403b4cbb8e78c7efd00153a3)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:06:50.652102 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfe26c5a-b13d-4b1d-b8f2-5551d6de55fc admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:06:50.660059 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=57 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:06:50.733301 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-bfe26c5a-b13d-4b1d-b8f2-5551d6de55fc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '07a6576d403b4cbb8e78c7efd00153a3'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:06:50.746929 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-bfe26c5a-b13d-4b1d-b8f2-5551d6de55fc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '07a6576d403b4cbb8e78c7efd00153a3'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:06:50.764477 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-bfe26c5a-b13d-4b1d-b8f2-5551d6de55fc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '07a6576d403b4cbb8e78c7efd00153a3'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:06:50.776453 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-bfe26c5a-b13d-4b1d-b8f2-5551d6de55fc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '07a6576d403b4cbb8e78c7efd00153a3'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:06:50.777680 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-bfe26c5a-b13d-4b1d-b8f2-5551d6de55fc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '07a6576d403b4cbb8e78c7efd00153a3'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:06:50.786233 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1006/2018] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:06:50 2022] DELETE /identity/v3/users/07a6576d403b4cbb8e78c7efd00153a3 => generated 0 bytes in 214 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:06:50.791798 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d27be953-ecde-445c-9fdf-26f20189c08f None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:50.792082 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d27be953-ecde-445c-9fdf-26f20189c08f None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:50.792354 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d27be953-ecde-445c-9fdf-26f20189c08f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:50.835838 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-d27be953-ecde-445c-9fdf-26f20189c08f None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:06:50.837766 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-d27be953-ecde-445c-9fdf-26f20189c08f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:06:50.930071 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1013/2019] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:06:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:06:50.940641 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfe26c5a-b13d-4b1d-b8f2-5551d6de55fc admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:06:50.946881 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-bfe26c5a-b13d-4b1d-b8f2-5551d6de55fc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:06:51.033166 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfe26c5a-b13d-4b1d-b8f2-5551d6de55fc admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:06:51.037985 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1611b71-9d60-4d89-9d3e-610257372e4e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:06:51.038621 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1611b71-9d60-4d89-9d3e-610257372e4e service neutron] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:51.042679 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1611b71-9d60-4d89-9d3e-610257372e4e service neutron] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:51.043244 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1611b71-9d60-4d89-9d3e-610257372e4e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:51.062790 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1611b71-9d60-4d89-9d3e-610257372e4e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:06:51.064493 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1611b71-9d60-4d89-9d3e-610257372e4e service neutron] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:06:51.077159 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1007/2020] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:06:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:06:51.406705 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d27be953-ecde-445c-9fdf-26f20189c08f None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:06:51.408140 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-d27be953-ecde-445c-9fdf-26f20189c08f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:06:51.421470 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d27be953-ecde-445c-9fdf-26f20189c08f None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:06:51.426181 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f34b1e9-b0b5-48ba-b29f-cccd09c7d644 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:06:51.426866 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f34b1e9-b0b5-48ba-b29f-cccd09c7d644 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:51.427152 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f34b1e9-b0b5-48ba-b29f-cccd09c7d644 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:51.427758 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f34b1e9-b0b5-48ba-b29f-cccd09c7d644 admin admin] PATH_INFO: `/v3/projects/9c6705155a634a18a0fda600155f6457` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:51.428796 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f34b1e9-b0b5-48ba-b29f-cccd09c7d644 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9c6705155a634a18a0fda600155f6457)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:06:51.431315 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f34b1e9-b0b5-48ba-b29f-cccd09c7d644 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:06:51.482462 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-6f34b1e9-b0b5-48ba-b29f-cccd09c7d644 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9c6705155a634a18a0fda600155f6457', 'request_id': 'req-6f34b1e9-b0b5-48ba-b29f-cccd09c7d644'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:06:51.488792 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-6f34b1e9-b0b5-48ba-b29f-cccd09c7d644 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9c6705155a634a18a0fda600155f6457', 'request_id': 'req-6f34b1e9-b0b5-48ba-b29f-cccd09c7d644'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:06:51.494281 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-6f34b1e9-b0b5-48ba-b29f-cccd09c7d644 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9c6705155a634a18a0fda600155f6457', 'request_id': 'req-6f34b1e9-b0b5-48ba-b29f-cccd09c7d644'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:06:51.506650 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-6f34b1e9-b0b5-48ba-b29f-cccd09c7d644 admin admin] The token cache is being invalidate because project 9c6705155a634a18a0fda600155f6457 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81561) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:06:51.506941 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-6f34b1e9-b0b5-48ba-b29f-cccd09c7d644 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:06:51.507548 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-6f34b1e9-b0b5-48ba-b29f-cccd09c7d644 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:06:51.509013 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1014/2021] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:06:51 2022] DELETE /identity/v3/projects/9c6705155a634a18a0fda600155f6457 => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:06:51.513030 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1611b71-9d60-4d89-9d3e-610257372e4e service neutron] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:06:51.522456 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1611b71-9d60-4d89-9d3e-610257372e4e service neutron] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:06:51.540253 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:06:51.585155 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb035127-91de-4f2f-9064-86ae94eed1d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:06:51.585776 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb035127-91de-4f2f-9064-86ae94eed1d5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:51.586054 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb035127-91de-4f2f-9064-86ae94eed1d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:51.586326 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb035127-91de-4f2f-9064-86ae94eed1d5 admin admin] PATH_INFO: `/v3/users/ce32c50e27754b65984f88ca1773f0c7` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:51.587463 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb035127-91de-4f2f-9064-86ae94eed1d5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ce32c50e27754b65984f88ca1773f0c7)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:06:51.589370 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb035127-91de-4f2f-9064-86ae94eed1d5 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:06:51.633874 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-eb035127-91de-4f2f-9064-86ae94eed1d5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ce32c50e27754b65984f88ca1773f0c7'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:06:51.642238 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-eb035127-91de-4f2f-9064-86ae94eed1d5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ce32c50e27754b65984f88ca1773f0c7'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:06:51.654746 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-eb035127-91de-4f2f-9064-86ae94eed1d5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ce32c50e27754b65984f88ca1773f0c7'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:06:51.661444 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-eb035127-91de-4f2f-9064-86ae94eed1d5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ce32c50e27754b65984f88ca1773f0c7'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:06:51.662069 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-eb035127-91de-4f2f-9064-86ae94eed1d5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ce32c50e27754b65984f88ca1773f0c7'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:06:51.667810 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1008/2022] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:06:51 2022] DELETE /identity/v3/users/ce32c50e27754b65984f88ca1773f0c7 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:06:51.938268 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f34b1e9-b0b5-48ba-b29f-cccd09c7d644 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:06:51.939816 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-6f34b1e9-b0b5-48ba-b29f-cccd09c7d644 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:06:51.954271 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f34b1e9-b0b5-48ba-b29f-cccd09c7d644 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:06:52.015669 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e89df0-f216-4445-9fb0-4f4f522de64a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:06:52.016307 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e89df0-f216-4445-9fb0-4f4f522de64a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:52.016587 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e89df0-f216-4445-9fb0-4f4f522de64a admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:52.016864 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e89df0-f216-4445-9fb0-4f4f522de64a admin admin] PATH_INFO: `/v3/projects/1a8ca1fdf8654505b50fc13ce94c598b` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:52.018864 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50e89df0-f216-4445-9fb0-4f4f522de64a admin admin] RBAC: Authorizing `identity:delete_project(project_id=1a8ca1fdf8654505b50fc13ce94c598b)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:06:52.020795 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50e89df0-f216-4445-9fb0-4f4f522de64a admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:06:52.052122 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:06:52.071327 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-50e89df0-f216-4445-9fb0-4f4f522de64a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1a8ca1fdf8654505b50fc13ce94c598b', 'request_id': 'req-50e89df0-f216-4445-9fb0-4f4f522de64a'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:06:52.078056 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-50e89df0-f216-4445-9fb0-4f4f522de64a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1a8ca1fdf8654505b50fc13ce94c598b', 'request_id': 'req-50e89df0-f216-4445-9fb0-4f4f522de64a'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:06:52.084006 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-50e89df0-f216-4445-9fb0-4f4f522de64a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1a8ca1fdf8654505b50fc13ce94c598b', 'request_id': 'req-50e89df0-f216-4445-9fb0-4f4f522de64a'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:06:52.098656 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-50e89df0-f216-4445-9fb0-4f4f522de64a admin admin] The token cache is being invalidate because project 1a8ca1fdf8654505b50fc13ce94c598b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81561) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:06:52.098961 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-50e89df0-f216-4445-9fb0-4f4f522de64a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:06:52.099623 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-50e89df0-f216-4445-9fb0-4f4f522de64a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:06:52.101452 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1015/2023] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:06:51 2022] DELETE /identity/v3/projects/1a8ca1fdf8654505b50fc13ce94c598b => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:06:52.106228 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5675f72-605d-4eca-b628-9871007de3f7 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:52.107253 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5675f72-605d-4eca-b628-9871007de3f7 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:52.107725 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5675f72-605d-4eca-b628-9871007de3f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:52.129392 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-c5675f72-605d-4eca-b628-9871007de3f7 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:06:52.130641 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-c5675f72-605d-4eca-b628-9871007de3f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:06:52.195919 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1009/2024] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:06:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:06:52.203451 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b6d791-39bd-4d1b-89dc-506a8e6e1a13 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:52.203738 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b6d791-39bd-4d1b-89dc-506a8e6e1a13 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:52.204015 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b6d791-39bd-4d1b-89dc-506a8e6e1a13 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:52.224188 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-d4b6d791-39bd-4d1b-89dc-506a8e6e1a13 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:06:52.226017 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-d4b6d791-39bd-4d1b-89dc-506a8e6e1a13 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:06:52.234446 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1016/2025] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:06:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:06:52.239039 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5675f72-605d-4eca-b628-9871007de3f7 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:06:52.250951 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5675f72-605d-4eca-b628-9871007de3f7 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:06:52.256494 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-580241a1-d20a-4153-91fb-c066d113ff94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:06:52.257163 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-580241a1-d20a-4153-91fb-c066d113ff94 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:52.257572 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-580241a1-d20a-4153-91fb-c066d113ff94 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:52.257851 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-580241a1-d20a-4153-91fb-c066d113ff94 admin admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:52.258521 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-580241a1-d20a-4153-91fb-c066d113ff94 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 09:06:52.259099 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-580241a1-d20a-4153-91fb-c066d113ff94 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:06:52.260883 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-580241a1-d20a-4153-91fb-c066d113ff94 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:06:52.278290 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1010/2026] 10.222.0.36 () {58 vars in 1153 bytes} [Thu Oct 20 09:06:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 09:06:52.279049 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4b6d791-39bd-4d1b-89dc-506a8e6e1a13 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:06:52.288539 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4b6d791-39bd-4d1b-89dc-506a8e6e1a13 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:06:52.293170 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e94c6f-3bb4-4cb6-9b5b-dda152e688f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:06:52.293769 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e94c6f-3bb4-4cb6-9b5b-dda152e688f3 admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:52.294045 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e94c6f-3bb4-4cb6-9b5b-dda152e688f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:52.294889 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e94c6f-3bb4-4cb6-9b5b-dda152e688f3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:52.295533 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6e94c6f-3bb4-4cb6-9b5b-dda152e688f3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:06:52.297482 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6e94c6f-3bb4-4cb6-9b5b-dda152e688f3 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:06:52.328242 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1017/2027] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:06:52 2022] POST /identity/v3/projects => generated 369 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 20 09:06:52.332444 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-580241a1-d20a-4153-91fb-c066d113ff94 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:06:52.343494 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-580241a1-d20a-4153-91fb-c066d113ff94 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:06:52.374587 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:06:52.408881 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f885c5d-7352-4704-91ac-c465a51742ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:06:52.409492 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f885c5d-7352-4704-91ac-c465a51742ea admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:52.409770 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f885c5d-7352-4704-91ac-c465a51742ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:52.410041 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f885c5d-7352-4704-91ac-c465a51742ea admin admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:52.410786 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f885c5d-7352-4704-91ac-c465a51742ea admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:06:52.412743 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f885c5d-7352-4704-91ac-c465a51742ea admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:06:52.435027 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1011/2028] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:06:52 2022] POST /identity/v3/users => generated 285 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:06:52.439225 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e94c6f-3bb4-4cb6-9b5b-dda152e688f3 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:06:52.450149 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e94c6f-3bb4-4cb6-9b5b-dda152e688f3 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:06:52.455080 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d702612e-e731-48b3-ae37-506bd567cab4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:06:52.455815 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d702612e-e731-48b3-ae37-506bd567cab4 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:52.456157 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d702612e-e731-48b3-ae37-506bd567cab4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:52.456496 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d702612e-e731-48b3-ae37-506bd567cab4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:52.457214 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d702612e-e731-48b3-ae37-506bd567cab4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:06:52.458976 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d702612e-e731-48b3-ae37-506bd567cab4 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:06:52.470508 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1018/2029] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:06:52 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 20 09:06:52.474693 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f885c5d-7352-4704-91ac-c465a51742ea admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:06:52.486101 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f885c5d-7352-4704-91ac-c465a51742ea admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:06:52.493214 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5076db2d-3313-4cd0-9815-7042912fc9bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:06:52.494341 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5076db2d-3313-4cd0-9815-7042912fc9bb admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:52.494721 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5076db2d-3313-4cd0-9815-7042912fc9bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:52.495067 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5076db2d-3313-4cd0-9815-7042912fc9bb admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:52.495842 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5076db2d-3313-4cd0-9815-7042912fc9bb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:06:52.497548 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5076db2d-3313-4cd0-9815-7042912fc9bb admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:06:52.508671 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1012/2030] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:06:52 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:06:52.512740 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d702612e-e731-48b3-ae37-506bd567cab4 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:06:52.523352 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d702612e-e731-48b3-ae37-506bd567cab4 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:06:52.528204 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-badca2dc-70f9-4c61-923f-3b0ff7f73f8c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:06:52.529574 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-badca2dc-70f9-4c61-923f-3b0ff7f73f8c admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:52.529867 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-badca2dc-70f9-4c61-923f-3b0ff7f73f8c admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:52.530213 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-badca2dc-70f9-4c61-923f-3b0ff7f73f8c admin admin] PATH_INFO: `/v3/projects/63e85e0fcff44e3694674fd085a26d8e/users/a11ea146ed384bada4f13e3a4a12cd18/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:52.545914 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-badca2dc-70f9-4c61-923f-3b0ff7f73f8c admin admin] RBAC: Authorizing `identity:create_grant(project_id=63e85e0fcff44e3694674fd085a26d8e, user_id=a11ea146ed384bada4f13e3a4a12cd18, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:06:52.548734 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-badca2dc-70f9-4c61-923f-3b0ff7f73f8c admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:06:52.562171 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1019/2031] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:06:52 2022] PUT /identity/v3/projects/63e85e0fcff44e3694674fd085a26d8e/users/a11ea146ed384bada4f13e3a4a12cd18/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:06:52.572513 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91567faf-e430-4ee0-bfb9-258ae1d77e82 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:52.572989 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91567faf-e430-4ee0-bfb9-258ae1d77e82 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:52.573086 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91567faf-e430-4ee0-bfb9-258ae1d77e82 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:52.601962 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-91567faf-e430-4ee0-bfb9-258ae1d77e82 None None] MFA Rules not processed for user `a11ea146ed384bada4f13e3a4a12cd18`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:06:52.603202 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-91567faf-e430-4ee0-bfb9-258ae1d77e82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:06:52.668285 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1013/2032] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:06:52 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:06:52.673136 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-badca2dc-70f9-4c61-923f-3b0ff7f73f8c admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:06:52.737325 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-badca2dc-70f9-4c61-923f-3b0ff7f73f8c admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:06:52.742198 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e9aaa4a-b1e5-4f38-b56b-c2571d3f9eb6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:06:52.742862 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e9aaa4a-b1e5-4f38-b56b-c2571d3f9eb6 admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:52.743133 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e9aaa4a-b1e5-4f38-b56b-c2571d3f9eb6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:52.743423 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e9aaa4a-b1e5-4f38-b56b-c2571d3f9eb6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:52.744089 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e9aaa4a-b1e5-4f38-b56b-c2571d3f9eb6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:06:52.746012 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e9aaa4a-b1e5-4f38-b56b-c2571d3f9eb6 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:06:52.773097 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1020/2033] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:06:52 2022] POST /identity/v3/projects => generated 369 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:06:52.777219 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91567faf-e430-4ee0-bfb9-258ae1d77e82 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:06:52.788227 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91567faf-e430-4ee0-bfb9-258ae1d77e82 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:06:52.843636 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90f06d52-a154-4e63-960c-16cd31570d8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:06:52.844322 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f06d52-a154-4e63-960c-16cd31570d8e admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:52.844598 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f06d52-a154-4e63-960c-16cd31570d8e admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:52.844863 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f06d52-a154-4e63-960c-16cd31570d8e admin admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:52.845478 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90f06d52-a154-4e63-960c-16cd31570d8e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:06:52.847589 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90f06d52-a154-4e63-960c-16cd31570d8e admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:06:52.868690 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1014/2034] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:06:52 2022] POST /identity/v3/users => generated 291 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:06:52.873300 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e9aaa4a-b1e5-4f38-b56b-c2571d3f9eb6 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:06:52.883446 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e9aaa4a-b1e5-4f38-b56b-c2571d3f9eb6 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:06:52.887928 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-806bfd1b-5ef6-45ea-b26e-dad5aa605732 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:06:52.888513 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-806bfd1b-5ef6-45ea-b26e-dad5aa605732 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:52.888786 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-806bfd1b-5ef6-45ea-b26e-dad5aa605732 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:52.889053 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-806bfd1b-5ef6-45ea-b26e-dad5aa605732 admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:52.889682 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-806bfd1b-5ef6-45ea-b26e-dad5aa605732 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:06:52.891298 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-806bfd1b-5ef6-45ea-b26e-dad5aa605732 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:06:52.901508 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1021/2035] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:06:52 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:06:52.905388 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90f06d52-a154-4e63-960c-16cd31570d8e admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:06:52.915204 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90f06d52-a154-4e63-960c-16cd31570d8e admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:06:52.920252 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fded249c-c3eb-4e4b-94ab-81a25ad5ee7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:06:52.920888 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fded249c-c3eb-4e4b-94ab-81a25ad5ee7e admin admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:52.921167 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fded249c-c3eb-4e4b-94ab-81a25ad5ee7e admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:52.921440 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fded249c-c3eb-4e4b-94ab-81a25ad5ee7e admin admin] PATH_INFO: `/v3/projects/6baa9519ed82409baf7a33f1d9b9b6f4/users/d8600fb8e95246ec8e7045b39e1f502c/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:52.936587 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fded249c-c3eb-4e4b-94ab-81a25ad5ee7e admin admin] RBAC: Authorizing `identity:create_grant(project_id=6baa9519ed82409baf7a33f1d9b9b6f4, user_id=d8600fb8e95246ec8e7045b39e1f502c, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:06:52.939280 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fded249c-c3eb-4e4b-94ab-81a25ad5ee7e admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:06:52.952768 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1015/2036] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:06:52 2022] PUT /identity/v3/projects/6baa9519ed82409baf7a33f1d9b9b6f4/users/d8600fb8e95246ec8e7045b39e1f502c/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:06:52.956632 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-806bfd1b-5ef6-45ea-b26e-dad5aa605732 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:06:52.967598 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-806bfd1b-5ef6-45ea-b26e-dad5aa605732 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:06:52.987243 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:DELETE=6,keystone:SELECT=90,keystone:INSERT=4 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:06:53.024920 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-183a03e5-99d8-4440-a969-3af88ffa6330 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:06:53.025514 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183a03e5-99d8-4440-a969-3af88ffa6330 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:53.025789 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183a03e5-99d8-4440-a969-3af88ffa6330 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:53.026055 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183a03e5-99d8-4440-a969-3af88ffa6330 admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:53.026737 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-183a03e5-99d8-4440-a969-3af88ffa6330 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:06:53.028337 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-183a03e5-99d8-4440-a969-3af88ffa6330 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:06:53.038447 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1022/2037] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:06:52 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:06:53.042453 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fded249c-c3eb-4e4b-94ab-81a25ad5ee7e admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:06:53.047774 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:06:53.054194 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fded249c-c3eb-4e4b-94ab-81a25ad5ee7e admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:06:53.059056 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09ba1386-c24a-40ce-9a4b-e0adddda0324 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:06:53.060194 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ba1386-c24a-40ce-9a4b-e0adddda0324 admin admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:53.060478 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ba1386-c24a-40ce-9a4b-e0adddda0324 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:53.060743 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ba1386-c24a-40ce-9a4b-e0adddda0324 admin admin] PATH_INFO: `/v3/projects/6baa9519ed82409baf7a33f1d9b9b6f4/users/d8600fb8e95246ec8e7045b39e1f502c/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:53.062627 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09ba1386-c24a-40ce-9a4b-e0adddda0324 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6baa9519ed82409baf7a33f1d9b9b6f4, user_id=d8600fb8e95246ec8e7045b39e1f502c, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:06:53.065372 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09ba1386-c24a-40ce-9a4b-e0adddda0324 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:06:53.075376 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1016/2038] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:06:53 2022] PUT /identity/v3/projects/6baa9519ed82409baf7a33f1d9b9b6f4/users/d8600fb8e95246ec8e7045b39e1f502c/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:06:53.086654 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec2911b2-94f5-413f-b981-732af38780f2 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:53.086938 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec2911b2-94f5-413f-b981-732af38780f2 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:53.087210 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec2911b2-94f5-413f-b981-732af38780f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:53.114489 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-ec2911b2-94f5-413f-b981-732af38780f2 None None] MFA Rules not processed for user `d8600fb8e95246ec8e7045b39e1f502c`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:06:53.115680 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-ec2911b2-94f5-413f-b981-732af38780f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:06:53.184532 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1023/2039] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:06:53 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:06:53.196343 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09ba1386-c24a-40ce-9a4b-e0adddda0324 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:06:53.198535 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-09ba1386-c24a-40ce-9a4b-e0adddda0324 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:06:53.256704 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09ba1386-c24a-40ce-9a4b-e0adddda0324 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:06:53.260363 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-298c51a0-72ab-4e4d-b985-e8fc2599cf94 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:06:53.261159 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-298c51a0-72ab-4e4d-b985-e8fc2599cf94 service cinder] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:53.261441 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-298c51a0-72ab-4e4d-b985-e8fc2599cf94 service cinder] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:53.261714 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-298c51a0-72ab-4e4d-b985-e8fc2599cf94 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:53.272025 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-298c51a0-72ab-4e4d-b985-e8fc2599cf94 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:06:53.273454 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-298c51a0-72ab-4e4d-b985-e8fc2599cf94 service cinder] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:06:53.284328 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1017/2040] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:06:53 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:06:53.329839 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec2911b2-94f5-413f-b981-732af38780f2 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:06:53.331278 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-ec2911b2-94f5-413f-b981-732af38780f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:06:53.344768 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec2911b2-94f5-413f-b981-732af38780f2 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:06:53.348633 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50384fda-95ed-4d12-8342-51806d9b6d23 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:06:53.349247 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50384fda-95ed-4d12-8342-51806d9b6d23 service cinder] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:53.349522 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50384fda-95ed-4d12-8342-51806d9b6d23 service cinder] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:53.349794 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50384fda-95ed-4d12-8342-51806d9b6d23 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:53.406892 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50384fda-95ed-4d12-8342-51806d9b6d23 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:06:53.408339 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50384fda-95ed-4d12-8342-51806d9b6d23 service cinder] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:06:53.417128 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1024/2041] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:06:53 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:06:56.741786 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-298c51a0-72ab-4e4d-b985-e8fc2599cf94 service cinder] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:06:56.823255 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-298c51a0-72ab-4e4d-b985-e8fc2599cf94 service cinder] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:06:56.827968 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a64fff4b-91d0-4459-800e-be90192f7703 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:06:56.828602 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a64fff4b-91d0-4459-800e-be90192f7703 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:56.828880 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a64fff4b-91d0-4459-800e-be90192f7703 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:56.829149 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a64fff4b-91d0-4459-800e-be90192f7703 admin admin] PATH_INFO: `/v3/users/a11ea146ed384bada4f13e3a4a12cd18` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:56.830267 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a64fff4b-91d0-4459-800e-be90192f7703 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a11ea146ed384bada4f13e3a4a12cd18)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:06:56.832213 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a64fff4b-91d0-4459-800e-be90192f7703 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:06:56.884164 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-a64fff4b-91d0-4459-800e-be90192f7703 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a11ea146ed384bada4f13e3a4a12cd18'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:06:56.892628 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-a64fff4b-91d0-4459-800e-be90192f7703 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a11ea146ed384bada4f13e3a4a12cd18'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:06:56.905164 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-a64fff4b-91d0-4459-800e-be90192f7703 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a11ea146ed384bada4f13e3a4a12cd18'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:06:56.912755 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-a64fff4b-91d0-4459-800e-be90192f7703 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a11ea146ed384bada4f13e3a4a12cd18'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:06:56.913507 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-a64fff4b-91d0-4459-800e-be90192f7703 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a11ea146ed384bada4f13e3a4a12cd18'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:06:56.920103 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1018/2042] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:06:56 2022] DELETE /identity/v3/users/a11ea146ed384bada4f13e3a4a12cd18 => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:06:56.924798 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a889f4-7d7b-495b-8b1f-61af5d015b4b None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:56.925290 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a889f4-7d7b-495b-8b1f-61af5d015b4b None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:56.925389 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a889f4-7d7b-495b-8b1f-61af5d015b4b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:56.946344 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-28a889f4-7d7b-495b-8b1f-61af5d015b4b None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:06:56.947651 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-28a889f4-7d7b-495b-8b1f-61af5d015b4b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:06:56.992502 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:06:57.007571 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df40e354-42d4-4b6d-ad30-490969c24553 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:57.007850 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df40e354-42d4-4b6d-ad30-490969c24553 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:57.008125 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df40e354-42d4-4b6d-ad30-490969c24553 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:57.009750 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1019/2043] 10.222.0.36 () {54 vars in 921 bytes} [Thu Oct 20 09:06:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 20 09:06:57.012337 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1025/2044] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:06:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 20 09:06:57.016947 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9218d22-4cd2-4525-9a08-280acc66d3ad None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:57.016947 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9218d22-4cd2-4525-9a08-280acc66d3ad None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:57.017217 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9218d22-4cd2-4525-9a08-280acc66d3ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:57.022607 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28a889f4-7d7b-495b-8b1f-61af5d015b4b None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:06:57.023979 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-28a889f4-7d7b-495b-8b1f-61af5d015b4b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:06:57.037259 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-b9218d22-4cd2-4525-9a08-280acc66d3ad None None] MFA Rules not processed for user `a94e868115ec49918eda10724263930c`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:06:57.038459 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-b9218d22-4cd2-4525-9a08-280acc66d3ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:06:57.044924 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:06:57.084134 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28a889f4-7d7b-495b-8b1f-61af5d015b4b None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:06:57.088232 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6727690-5e89-4603-a86d-f92935706467 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:06:57.088845 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6727690-5e89-4603-a86d-f92935706467 service neutron] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:57.089125 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6727690-5e89-4603-a86d-f92935706467 service neutron] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:57.089397 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6727690-5e89-4603-a86d-f92935706467 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:57.099462 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6727690-5e89-4603-a86d-f92935706467 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:06:57.100845 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6727690-5e89-4603-a86d-f92935706467 service neutron] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:06:57.114192 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1026/2045] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:06:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 20 09:06:57.120564 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1020/2046] 10.222.0.36 () {58 vars in 1018 bytes} [Thu Oct 20 09:06:57 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:06:57.132314 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6727690-5e89-4603-a86d-f92935706467 service neutron] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:06:57.133800 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-e6727690-5e89-4603-a86d-f92935706467 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:06:57.201525 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6727690-5e89-4603-a86d-f92935706467 service neutron] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:06:57.205548 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81aed70d-c593-4e8e-889d-e3586159fd2e service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce043914cd9042909f70ab0adb272d34', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:06:57.206167 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81aed70d-c593-4e8e-889d-e3586159fd2e service glance] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:57.206443 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81aed70d-c593-4e8e-889d-e3586159fd2e service glance] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:57.206742 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81aed70d-c593-4e8e-889d-e3586159fd2e service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:57.218105 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81aed70d-c593-4e8e-889d-e3586159fd2e service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:06:57.219555 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81aed70d-c593-4e8e-889d-e3586159fd2e service glance] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:06:57.230890 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1027/2047] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:06:57 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 20 09:06:57.303844 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0fe4426-e1d8-4064-af47-759b96befe23 None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:57.304129 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0fe4426-e1d8-4064-af47-759b96befe23 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:57.304400 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0fe4426-e1d8-4064-af47-759b96befe23 None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:57.305948 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1021/2048] 10.222.0.36 () {54 vars in 921 bytes} [Thu Oct 20 09:06:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 20 09:06:57.311158 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5733bfc9-e144-4043-ab2b-be3059c5b635 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:57.311621 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5733bfc9-e144-4043-ab2b-be3059c5b635 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:57.311889 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5733bfc9-e144-4043-ab2b-be3059c5b635 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:57.333044 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-5733bfc9-e144-4043-ab2b-be3059c5b635 None None] MFA Rules not processed for user `794ca389002d41748d7925216ed964a5`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:06:57.334378 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-5733bfc9-e144-4043-ab2b-be3059c5b635 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:06:57.341859 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1028/2049] 10.222.0.36 () {58 vars in 1018 bytes} [Thu Oct 20 09:06:57 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:06:57.353795 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0fe4426-e1d8-4064-af47-759b96befe23 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:06:57.355618 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-a0fe4426-e1d8-4064-af47-759b96befe23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:06:57.370148 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0fe4426-e1d8-4064-af47-759b96befe23 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:06:57.374374 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b603b1-5c46-4810-9ff6-b7890716db0e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:06:57.375034 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b603b1-5c46-4810-9ff6-b7890716db0e service neutron] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:57.375314 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b603b1-5c46-4810-9ff6-b7890716db0e service neutron] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:57.375584 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b603b1-5c46-4810-9ff6-b7890716db0e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:57.383599 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1b603b1-5c46-4810-9ff6-b7890716db0e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:06:57.384496 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1b603b1-5c46-4810-9ff6-b7890716db0e service neutron] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:06:57.392984 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1022/2050] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:06:57 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:06:57.431092 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5733bfc9-e144-4043-ab2b-be3059c5b635 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:06:57.433006 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-5733bfc9-e144-4043-ab2b-be3059c5b635 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:06:57.448857 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5733bfc9-e144-4043-ab2b-be3059c5b635 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:06:57.464024 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b26c13f-342b-4c41-9e6e-9c104620f9c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:06:57.465356 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b26c13f-342b-4c41-9e6e-9c104620f9c9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:57.467145 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b26c13f-342b-4c41-9e6e-9c104620f9c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:57.467700 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b26c13f-342b-4c41-9e6e-9c104620f9c9 admin admin] PATH_INFO: `/v3/projects/63e85e0fcff44e3694674fd085a26d8e` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:57.469622 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b26c13f-342b-4c41-9e6e-9c104620f9c9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=63e85e0fcff44e3694674fd085a26d8e)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:06:57.473110 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b26c13f-342b-4c41-9e6e-9c104620f9c9 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:06:57.495710 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e18882-0762-4d9f-b91e-3885b15d916b None None] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:57.495994 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e18882-0762-4d9f-b91e-3885b15d916b None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:57.496305 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e18882-0762-4d9f-b91e-3885b15d916b None None] PATH_INFO: `/` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:57.497811 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1023/2051] 10.222.0.36 () {54 vars in 921 bytes} [Thu Oct 20 09:06:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 20 09:06:57.503928 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed1fb88-ca22-4bdd-9ff0-cd1fc2e8bffe None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:57.504397 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed1fb88-ca22-4bdd-9ff0-cd1fc2e8bffe None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:57.504690 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed1fb88-ca22-4bdd-9ff0-cd1fc2e8bffe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:57.526442 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-fed1fb88-ca22-4bdd-9ff0-cd1fc2e8bffe None None] MFA Rules not processed for user `a94e868115ec49918eda10724263930c`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:06:57.526668 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-3b26c13f-342b-4c41-9e6e-9c104620f9c9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '63e85e0fcff44e3694674fd085a26d8e', 'request_id': 'req-3b26c13f-342b-4c41-9e6e-9c104620f9c9'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:06:57.527876 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-fed1fb88-ca22-4bdd-9ff0-cd1fc2e8bffe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:06:57.533917 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-3b26c13f-342b-4c41-9e6e-9c104620f9c9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '63e85e0fcff44e3694674fd085a26d8e', 'request_id': 'req-3b26c13f-342b-4c41-9e6e-9c104620f9c9'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:06:57.540000 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-3b26c13f-342b-4c41-9e6e-9c104620f9c9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '63e85e0fcff44e3694674fd085a26d8e', 'request_id': 'req-3b26c13f-342b-4c41-9e6e-9c104620f9c9'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:06:57.553240 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-3b26c13f-342b-4c41-9e6e-9c104620f9c9 admin admin] The token cache is being invalidate because project 63e85e0fcff44e3694674fd085a26d8e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81561) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:06:57.553522 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-3b26c13f-342b-4c41-9e6e-9c104620f9c9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:06:57.554152 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-3b26c13f-342b-4c41-9e6e-9c104620f9c9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:06:57.555945 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1029/2052] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:06:57 2022] DELETE /identity/v3/projects/63e85e0fcff44e3694674fd085a26d8e => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:06:57.559630 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b26c13f-342b-4c41-9e6e-9c104620f9c9 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:06:57.561490 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-3b26c13f-342b-4c41-9e6e-9c104620f9c9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:06:57.571269 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:06:57.599915 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1024/2053] 10.222.0.36 () {58 vars in 1018 bytes} [Thu Oct 20 09:06:57 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 20 09:06:57.611020 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed1fb88-ca22-4bdd-9ff0-cd1fc2e8bffe None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:06:57.612383 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-fed1fb88-ca22-4bdd-9ff0-cd1fc2e8bffe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:06:57.632678 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b26c13f-342b-4c41-9e6e-9c104620f9c9 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:06:57.637304 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a352707-0067-4587-aa45-2f8a4e736d82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:06:57.637966 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a352707-0067-4587-aa45-2f8a4e736d82 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:57.638236 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a352707-0067-4587-aa45-2f8a4e736d82 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:57.638520 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a352707-0067-4587-aa45-2f8a4e736d82 admin admin] PATH_INFO: `/v3/users/d8600fb8e95246ec8e7045b39e1f502c` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:57.639650 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a352707-0067-4587-aa45-2f8a4e736d82 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d8600fb8e95246ec8e7045b39e1f502c)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:06:57.641552 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a352707-0067-4587-aa45-2f8a4e736d82 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:06:57.668659 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed1fb88-ca22-4bdd-9ff0-cd1fc2e8bffe None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:06:57.673473 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b88b0c82-c9cb-41fa-a491-4337c21460da service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:06:57.674324 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88b0c82-c9cb-41fa-a491-4337c21460da service neutron] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:57.674593 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88b0c82-c9cb-41fa-a491-4337c21460da service neutron] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:57.674864 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88b0c82-c9cb-41fa-a491-4337c21460da service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:57.685666 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b88b0c82-c9cb-41fa-a491-4337c21460da service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:06:57.687148 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b88b0c82-c9cb-41fa-a491-4337c21460da service neutron] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:06:57.688869 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-4a352707-0067-4587-aa45-2f8a4e736d82 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd8600fb8e95246ec8e7045b39e1f502c'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:06:57.698218 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1025/2054] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:06:57 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 20 09:06:57.698624 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-4a352707-0067-4587-aa45-2f8a4e736d82 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd8600fb8e95246ec8e7045b39e1f502c'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:06:57.712767 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-4a352707-0067-4587-aa45-2f8a4e736d82 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd8600fb8e95246ec8e7045b39e1f502c'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:06:57.720464 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-4a352707-0067-4587-aa45-2f8a4e736d82 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd8600fb8e95246ec8e7045b39e1f502c'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:06:57.721181 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-4a352707-0067-4587-aa45-2f8a4e736d82 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd8600fb8e95246ec8e7045b39e1f502c'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:06:57.726965 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1030/2055] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:06:57 2022] DELETE /identity/v3/users/d8600fb8e95246ec8e7045b39e1f502c => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:06:58.075080 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b88b0c82-c9cb-41fa-a491-4337c21460da service neutron] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:06:58.091445 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b88b0c82-c9cb-41fa-a491-4337c21460da service neutron] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:06:58.129060 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:06:58.176217 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89e372be-6ea4-4bd8-a744-4653578f9408 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:06:58.176960 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e372be-6ea4-4bd8-a744-4653578f9408 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:58.177363 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e372be-6ea4-4bd8-a744-4653578f9408 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:58.177687 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e372be-6ea4-4bd8-a744-4653578f9408 admin admin] PATH_INFO: `/v3/projects/6baa9519ed82409baf7a33f1d9b9b6f4` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:58.179357 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89e372be-6ea4-4bd8-a744-4653578f9408 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6baa9519ed82409baf7a33f1d9b9b6f4)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:06:58.181971 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89e372be-6ea4-4bd8-a744-4653578f9408 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:06:58.246700 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-89e372be-6ea4-4bd8-a744-4653578f9408 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6baa9519ed82409baf7a33f1d9b9b6f4', 'request_id': 'req-89e372be-6ea4-4bd8-a744-4653578f9408'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:06:58.254324 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-89e372be-6ea4-4bd8-a744-4653578f9408 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6baa9519ed82409baf7a33f1d9b9b6f4', 'request_id': 'req-89e372be-6ea4-4bd8-a744-4653578f9408'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:06:58.264307 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-89e372be-6ea4-4bd8-a744-4653578f9408 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6baa9519ed82409baf7a33f1d9b9b6f4', 'request_id': 'req-89e372be-6ea4-4bd8-a744-4653578f9408'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:06:58.277675 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-89e372be-6ea4-4bd8-a744-4653578f9408 admin admin] The token cache is being invalidate because project 6baa9519ed82409baf7a33f1d9b9b6f4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81562) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:06:58.277960 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-89e372be-6ea4-4bd8-a744-4653578f9408 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:06:58.279678 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-89e372be-6ea4-4bd8-a744-4653578f9408 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:06:58.280793 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1026/2056] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:06:58 2022] DELETE /identity/v3/projects/6baa9519ed82409baf7a33f1d9b9b6f4 => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:06:58.285699 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c104b75d-0974-4c0f-a685-ac6508600d5d None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:58.286200 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c104b75d-0974-4c0f-a685-ac6508600d5d None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:58.286301 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c104b75d-0974-4c0f-a685-ac6508600d5d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:58.310444 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-c104b75d-0974-4c0f-a685-ac6508600d5d None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:06:58.311769 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-c104b75d-0974-4c0f-a685-ac6508600d5d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:06:58.384463 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1031/2057] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:06:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:06:58.386410 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8876b73d-7835-4daf-84d9-614aeb9cd033 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:58.386961 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8876b73d-7835-4daf-84d9-614aeb9cd033 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:58.387235 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8876b73d-7835-4daf-84d9-614aeb9cd033 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:58.408082 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-8876b73d-7835-4daf-84d9-614aeb9cd033 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:06:58.409284 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-8876b73d-7835-4daf-84d9-614aeb9cd033 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:06:58.417398 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1027/2058] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:06:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:06:58.421625 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c104b75d-0974-4c0f-a685-ac6508600d5d None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:06:58.431602 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c104b75d-0974-4c0f-a685-ac6508600d5d None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:06:58.436971 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c8c0ad1-f3a6-47f2-9228-194a5b7be9f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:06:58.437581 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8c0ad1-f3a6-47f2-9228-194a5b7be9f2 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:58.437873 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8c0ad1-f3a6-47f2-9228-194a5b7be9f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:58.438139 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8c0ad1-f3a6-47f2-9228-194a5b7be9f2 admin admin] PATH_INFO: `/v3/domains` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:58.438790 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c8c0ad1-f3a6-47f2-9228-194a5b7be9f2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 09:06:58.439085 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c8c0ad1-f3a6-47f2-9228-194a5b7be9f2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:06:58.440675 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c8c0ad1-f3a6-47f2-9228-194a5b7be9f2 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:06:58.453915 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1032/2059] 10.222.0.36 () {58 vars in 1153 bytes} [Thu Oct 20 09:06:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 09:06:58.458069 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8876b73d-7835-4daf-84d9-614aeb9cd033 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:06:58.468517 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8876b73d-7835-4daf-84d9-614aeb9cd033 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:06:58.473742 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c200315-76b1-443a-b9fd-e87fa8b48192 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:06:58.474333 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c200315-76b1-443a-b9fd-e87fa8b48192 admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:58.474644 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c200315-76b1-443a-b9fd-e87fa8b48192 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:58.474916 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c200315-76b1-443a-b9fd-e87fa8b48192 admin admin] PATH_INFO: `/v3/projects` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:58.475545 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c200315-76b1-443a-b9fd-e87fa8b48192 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:06:58.477428 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c200315-76b1-443a-b9fd-e87fa8b48192 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:06:58.511658 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1028/2060] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:06:58 2022] POST /identity/v3/projects => generated 391 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:06:58.512497 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c8c0ad1-f3a6-47f2-9228-194a5b7be9f2 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:06:58.523779 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c8c0ad1-f3a6-47f2-9228-194a5b7be9f2 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:06:58.532318 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:06:58.581681 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816a29c1-10a3-472f-b071-2760d98b51e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:06:58.582282 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816a29c1-10a3-472f-b071-2760d98b51e0 admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:58.582583 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816a29c1-10a3-472f-b071-2760d98b51e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:58.582852 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816a29c1-10a3-472f-b071-2760d98b51e0 admin admin] PATH_INFO: `/v3/users` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:58.583493 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-816a29c1-10a3-472f-b071-2760d98b51e0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:06:58.585404 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-816a29c1-10a3-472f-b071-2760d98b51e0 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:06:58.608815 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1033/2061] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:06:58 2022] POST /identity/v3/users => generated 296 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:06:58.613124 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c200315-76b1-443a-b9fd-e87fa8b48192 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:06:58.623166 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c200315-76b1-443a-b9fd-e87fa8b48192 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:06:58.627657 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f9e06f1-f579-44ca-9d71-3a38390031c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:06:58.628244 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f9e06f1-f579-44ca-9d71-3a38390031c8 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:58.628518 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f9e06f1-f579-44ca-9d71-3a38390031c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:58.628871 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f9e06f1-f579-44ca-9d71-3a38390031c8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:58.629399 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f9e06f1-f579-44ca-9d71-3a38390031c8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:06:58.631075 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f9e06f1-f579-44ca-9d71-3a38390031c8 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:06:58.640986 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1029/2062] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:06:58 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:06:58.645111 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816a29c1-10a3-472f-b071-2760d98b51e0 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:06:58.655584 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816a29c1-10a3-472f-b071-2760d98b51e0 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:06:58.660188 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50433d32-2f9c-4a3c-8611-c81950643e4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:06:58.660786 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50433d32-2f9c-4a3c-8611-c81950643e4d admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:58.661062 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50433d32-2f9c-4a3c-8611-c81950643e4d admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:58.661329 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50433d32-2f9c-4a3c-8611-c81950643e4d admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:58.662096 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50433d32-2f9c-4a3c-8611-c81950643e4d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:06:58.663734 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50433d32-2f9c-4a3c-8611-c81950643e4d admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:06:58.675020 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1034/2063] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:06:58 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:06:58.679218 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f9e06f1-f579-44ca-9d71-3a38390031c8 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:06:58.689058 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f9e06f1-f579-44ca-9d71-3a38390031c8 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:06:58.693678 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0194a029-2cb0-427e-a7d7-b45167c1ef1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:06:58.694338 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0194a029-2cb0-427e-a7d7-b45167c1ef1e admin admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:58.694655 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0194a029-2cb0-427e-a7d7-b45167c1ef1e admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:58.694928 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0194a029-2cb0-427e-a7d7-b45167c1ef1e admin admin] PATH_INFO: `/v3/projects/27435bf6f48543a5be061a248c79eafd/users/d20e533a3a3845e288b95b7909e7c45c/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:58.710446 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0194a029-2cb0-427e-a7d7-b45167c1ef1e admin admin] RBAC: Authorizing `identity:create_grant(project_id=27435bf6f48543a5be061a248c79eafd, user_id=d20e533a3a3845e288b95b7909e7c45c, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:06:58.713086 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0194a029-2cb0-427e-a7d7-b45167c1ef1e admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:06:58.726894 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1030/2064] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:06:58 2022] PUT /identity/v3/projects/27435bf6f48543a5be061a248c79eafd/users/d20e533a3a3845e288b95b7909e7c45c/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:06:58.738260 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3979ee9d-93c8-41e1-abf1-486293e07820 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:58.738659 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3979ee9d-93c8-41e1-abf1-486293e07820 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:58.739001 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3979ee9d-93c8-41e1-abf1-486293e07820 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:58.768084 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-3979ee9d-93c8-41e1-abf1-486293e07820 None None] MFA Rules not processed for user `d20e533a3a3845e288b95b7909e7c45c`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:06:58.769300 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-3979ee9d-93c8-41e1-abf1-486293e07820 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:06:58.833605 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1035/2065] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:06:58 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:06:58.838612 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0194a029-2cb0-427e-a7d7-b45167c1ef1e admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:06:58.892634 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:06:58.906019 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0194a029-2cb0-427e-a7d7-b45167c1ef1e admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:06:58.911166 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-205d703a-c3af-4c7c-ae0b-7f6fd2531a22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:06:58.911767 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-205d703a-c3af-4c7c-ae0b-7f6fd2531a22 admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:58.912044 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-205d703a-c3af-4c7c-ae0b-7f6fd2531a22 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:58.912313 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-205d703a-c3af-4c7c-ae0b-7f6fd2531a22 admin admin] PATH_INFO: `/v3/projects` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:58.912945 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-205d703a-c3af-4c7c-ae0b-7f6fd2531a22 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:06:58.914910 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-205d703a-c3af-4c7c-ae0b-7f6fd2531a22 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:06:58.945716 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1031/2066] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:06:58 2022] POST /identity/v3/projects => generated 391 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 20 09:06:58.949795 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3979ee9d-93c8-41e1-abf1-486293e07820 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:06:58.963344 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3979ee9d-93c8-41e1-abf1-486293e07820 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:06:59.022886 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2c0180f-cd1c-4042-a57e-5bcf33503f50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:06:59.023498 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2c0180f-cd1c-4042-a57e-5bcf33503f50 admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:59.023776 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2c0180f-cd1c-4042-a57e-5bcf33503f50 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:59.024047 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2c0180f-cd1c-4042-a57e-5bcf33503f50 admin admin] PATH_INFO: `/v3/users` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:59.024708 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2c0180f-cd1c-4042-a57e-5bcf33503f50 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:06:59.026693 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2c0180f-cd1c-4042-a57e-5bcf33503f50 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:06:59.050392 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1036/2067] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:06:58 2022] POST /identity/v3/users => generated 302 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:06:59.054152 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-205d703a-c3af-4c7c-ae0b-7f6fd2531a22 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:06:59.065474 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-205d703a-c3af-4c7c-ae0b-7f6fd2531a22 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:06:59.070225 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f5450ad-daad-41d0-887e-a88f00a8faed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:06:59.070850 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f5450ad-daad-41d0-887e-a88f00a8faed admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:59.071127 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f5450ad-daad-41d0-887e-a88f00a8faed admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:59.071424 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f5450ad-daad-41d0-887e-a88f00a8faed admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:59.072062 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f5450ad-daad-41d0-887e-a88f00a8faed admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:06:59.073740 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f5450ad-daad-41d0-887e-a88f00a8faed admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:06:59.085071 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1032/2068] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:06:59 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:06:59.089282 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2c0180f-cd1c-4042-a57e-5bcf33503f50 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:06:59.099914 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2c0180f-cd1c-4042-a57e-5bcf33503f50 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:06:59.104396 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f548c49c-8654-41af-a5e6-204b44834df2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:06:59.105047 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f548c49c-8654-41af-a5e6-204b44834df2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:59.105323 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f548c49c-8654-41af-a5e6-204b44834df2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:59.105590 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f548c49c-8654-41af-a5e6-204b44834df2 admin admin] PATH_INFO: `/v3/projects/b92714ece8074d50ac508ddccd193850/users/26eb43af36b74072943b75c446ebd613/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:59.115574 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:06:59.123493 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f548c49c-8654-41af-a5e6-204b44834df2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b92714ece8074d50ac508ddccd193850, user_id=26eb43af36b74072943b75c446ebd613, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:06:59.126241 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f548c49c-8654-41af-a5e6-204b44834df2 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:06:59.141020 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1037/2069] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:06:59 2022] PUT /identity/v3/projects/b92714ece8074d50ac508ddccd193850/users/26eb43af36b74072943b75c446ebd613/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:06:59.145118 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f5450ad-daad-41d0-887e-a88f00a8faed admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:06:59.157804 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f5450ad-daad-41d0-887e-a88f00a8faed admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:06:59.226497 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c461f381-4ece-4fa8-b93a-49bd709be0a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:06:59.227044 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c461f381-4ece-4fa8-b93a-49bd709be0a6 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:59.227323 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c461f381-4ece-4fa8-b93a-49bd709be0a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:59.227593 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c461f381-4ece-4fa8-b93a-49bd709be0a6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:59.228233 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c461f381-4ece-4fa8-b93a-49bd709be0a6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:06:59.229832 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c461f381-4ece-4fa8-b93a-49bd709be0a6 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:06:59.241202 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1033/2070] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:06:59 2022] GET /identity/v3/roles => generated 1420 bytes in 97 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:06:59.245429 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f548c49c-8654-41af-a5e6-204b44834df2 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:06:59.260058 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f548c49c-8654-41af-a5e6-204b44834df2 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:06:59.266609 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c10da7-d8cd-443c-85f7-b1f934f2686b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:06:59.267349 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c10da7-d8cd-443c-85f7-b1f934f2686b admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:59.267717 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c10da7-d8cd-443c-85f7-b1f934f2686b admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:59.268056 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c10da7-d8cd-443c-85f7-b1f934f2686b admin admin] PATH_INFO: `/v3/projects/b92714ece8074d50ac508ddccd193850/users/26eb43af36b74072943b75c446ebd613/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:59.270013 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5c10da7-d8cd-443c-85f7-b1f934f2686b admin admin] RBAC: Authorizing `identity:create_grant(project_id=b92714ece8074d50ac508ddccd193850, user_id=26eb43af36b74072943b75c446ebd613, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:06:59.272752 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5c10da7-d8cd-443c-85f7-b1f934f2686b admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:06:59.286535 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1038/2071] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:06:59 2022] PUT /identity/v3/projects/b92714ece8074d50ac508ddccd193850/users/26eb43af36b74072943b75c446ebd613/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 42 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:06:59.296945 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c42636-ba07-4a6d-9351-da3e82af2adb None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:59.297237 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c42636-ba07-4a6d-9351-da3e82af2adb None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:59.297520 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c42636-ba07-4a6d-9351-da3e82af2adb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:59.328366 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-04c42636-ba07-4a6d-9351-da3e82af2adb None None] MFA Rules not processed for user `26eb43af36b74072943b75c446ebd613`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:06:59.329799 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-04c42636-ba07-4a6d-9351-da3e82af2adb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:06:59.409134 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1034/2072] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:06:59 2022] POST /identity/v3/auth/tokens => generated 3335 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:06:59.422348 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c10da7-d8cd-443c-85f7-b1f934f2686b admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:06:59.424409 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-c5c10da7-d8cd-443c-85f7-b1f934f2686b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:06:59.490614 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c10da7-d8cd-443c-85f7-b1f934f2686b admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:06:59.494317 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f410d4-2093-42d0-8168-637e59a595b1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:06:59.494962 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f410d4-2093-42d0-8168-637e59a595b1 service cinder] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:59.495251 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f410d4-2093-42d0-8168-637e59a595b1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:59.495511 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f410d4-2093-42d0-8168-637e59a595b1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:59.505297 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6f410d4-2093-42d0-8168-637e59a595b1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:06:59.506774 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6f410d4-2093-42d0-8168-637e59a595b1 service cinder] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:06:59.516803 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1039/2073] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:06:59 2022] GET /identity/v3/auth/tokens => generated 3335 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:06:59.563563 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04c42636-ba07-4a6d-9351-da3e82af2adb None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:06:59.564986 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-04c42636-ba07-4a6d-9351-da3e82af2adb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:06:59.578434 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04c42636-ba07-4a6d-9351-da3e82af2adb None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:06:59.582221 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c1df99d-be1f-41f7-821f-e0797ba51bed service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:06:59.582853 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c1df99d-be1f-41f7-821f-e0797ba51bed service cinder] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:59.583177 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c1df99d-be1f-41f7-821f-e0797ba51bed service cinder] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:59.583497 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c1df99d-be1f-41f7-821f-e0797ba51bed service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:59.630294 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:06:59.655988 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c1df99d-be1f-41f7-821f-e0797ba51bed service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:06:59.657438 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c1df99d-be1f-41f7-821f-e0797ba51bed service cinder] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:06:59.667325 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1035/2074] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:06:59 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:06:59.892107 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60ccc757-d446-49fa-ae4e-5683f8de5fe5 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:06:59.892600 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60ccc757-d446-49fa-ae4e-5683f8de5fe5 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:06:59.892877 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60ccc757-d446-49fa-ae4e-5683f8de5fe5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:06:59.914364 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-60ccc757-d446-49fa-ae4e-5683f8de5fe5 None None] MFA Rules not processed for user `f4b0e9187ce74f7888e9e7094be5c824`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:06:59.915574 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-60ccc757-d446-49fa-ae4e-5683f8de5fe5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:07:00.009348 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1040/2075] 10.222.0.36 () {58 vars in 1009 bytes} [Thu Oct 20 09:06:59 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:07:00.024042 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c1df99d-be1f-41f7-821f-e0797ba51bed service cinder] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:00.025845 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-4c1df99d-be1f-41f7-821f-e0797ba51bed service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:07:00.103034 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c1df99d-be1f-41f7-821f-e0797ba51bed service cinder] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:00.108794 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c7aee1a-a269-4b03-8cd1-d3d51491a322 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4d558918cd8b4ebea4fe8765297304db', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:00.109474 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c7aee1a-a269-4b03-8cd1-d3d51491a322 service swift] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:00.109830 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c7aee1a-a269-4b03-8cd1-d3d51491a322 service swift] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:00.110570 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c7aee1a-a269-4b03-8cd1-d3d51491a322 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:00.121294 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c7aee1a-a269-4b03-8cd1-d3d51491a322 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:00.123168 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c7aee1a-a269-4b03-8cd1-d3d51491a322 service swift] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:00.132302 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1036/2076] 10.222.0.36 () {60 vars in 1384 bytes} [Thu Oct 20 09:07:00 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 20 09:07:03.128728 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60ccc757-d446-49fa-ae4e-5683f8de5fe5 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:03.215499 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60ccc757-d446-49fa-ae4e-5683f8de5fe5 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:03.221165 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-205bfd62-f5f2-472a-9026-c06911601478 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:03.221897 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-205bfd62-f5f2-472a-9026-c06911601478 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:03.222183 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-205bfd62-f5f2-472a-9026-c06911601478 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:03.222445 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-205bfd62-f5f2-472a-9026-c06911601478 admin admin] PATH_INFO: `/v3/users/d20e533a3a3845e288b95b7909e7c45c` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:03.223609 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-205bfd62-f5f2-472a-9026-c06911601478 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d20e533a3a3845e288b95b7909e7c45c)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:03.225654 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-205bfd62-f5f2-472a-9026-c06911601478 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:03.252532 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=94,keystone:INSERT=2,keystone:DELETE=4 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:07:03.290049 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-205bfd62-f5f2-472a-9026-c06911601478 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd20e533a3a3845e288b95b7909e7c45c'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:03.297796 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-205bfd62-f5f2-472a-9026-c06911601478 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd20e533a3a3845e288b95b7909e7c45c'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:03.310779 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-205bfd62-f5f2-472a-9026-c06911601478 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd20e533a3a3845e288b95b7909e7c45c'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:03.318529 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-205bfd62-f5f2-472a-9026-c06911601478 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd20e533a3a3845e288b95b7909e7c45c'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:03.319421 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-205bfd62-f5f2-472a-9026-c06911601478 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd20e533a3a3845e288b95b7909e7c45c'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:03.326607 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1041/2077] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:07:03 2022] DELETE /identity/v3/users/d20e533a3a3845e288b95b7909e7c45c => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:07:03.331114 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c324e753-eeed-4b1f-8e9e-58814f428cf0 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:03.331700 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c324e753-eeed-4b1f-8e9e-58814f428cf0 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:03.331700 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c324e753-eeed-4b1f-8e9e-58814f428cf0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:03.351987 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-c324e753-eeed-4b1f-8e9e-58814f428cf0 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:07:03.353179 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-c324e753-eeed-4b1f-8e9e-58814f428cf0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:07:03.420084 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1037/2078] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:07:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:07:03.430230 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-205bfd62-f5f2-472a-9026-c06911601478 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:03.431642 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-205bfd62-f5f2-472a-9026-c06911601478 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:07:03.498558 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-205bfd62-f5f2-472a-9026-c06911601478 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:03.502325 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5412d89a-949b-4d6e-a02b-337b155a1609 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:03.502975 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5412d89a-949b-4d6e-a02b-337b155a1609 service neutron] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:03.503261 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5412d89a-949b-4d6e-a02b-337b155a1609 service neutron] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:03.503539 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5412d89a-949b-4d6e-a02b-337b155a1609 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:03.513876 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5412d89a-949b-4d6e-a02b-337b155a1609 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:03.515359 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5412d89a-949b-4d6e-a02b-337b155a1609 service neutron] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:03.526388 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1042/2079] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:07:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:07:03.795562 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c324e753-eeed-4b1f-8e9e-58814f428cf0 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:03.796999 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-c324e753-eeed-4b1f-8e9e-58814f428cf0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:07:03.827029 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c324e753-eeed-4b1f-8e9e-58814f428cf0 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:03.847172 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a6c0125-10a5-49e5-a431-7f38e44594f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:03.847827 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a6c0125-10a5-49e5-a431-7f38e44594f0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:03.848107 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a6c0125-10a5-49e5-a431-7f38e44594f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:03.848380 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a6c0125-10a5-49e5-a431-7f38e44594f0 admin admin] PATH_INFO: `/v3/projects/27435bf6f48543a5be061a248c79eafd` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:03.849512 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a6c0125-10a5-49e5-a431-7f38e44594f0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=27435bf6f48543a5be061a248c79eafd)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:03.851559 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a6c0125-10a5-49e5-a431-7f38e44594f0 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:03.901942 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-3a6c0125-10a5-49e5-a431-7f38e44594f0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '27435bf6f48543a5be061a248c79eafd', 'request_id': 'req-3a6c0125-10a5-49e5-a431-7f38e44594f0'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:03.909692 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-3a6c0125-10a5-49e5-a431-7f38e44594f0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '27435bf6f48543a5be061a248c79eafd', 'request_id': 'req-3a6c0125-10a5-49e5-a431-7f38e44594f0'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:03.915667 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-3a6c0125-10a5-49e5-a431-7f38e44594f0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '27435bf6f48543a5be061a248c79eafd', 'request_id': 'req-3a6c0125-10a5-49e5-a431-7f38e44594f0'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:03.924717 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:07:03.929472 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-3a6c0125-10a5-49e5-a431-7f38e44594f0 admin admin] The token cache is being invalidate because project 27435bf6f48543a5be061a248c79eafd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81562) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:07:03.929773 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-3a6c0125-10a5-49e5-a431-7f38e44594f0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:03.930888 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-3a6c0125-10a5-49e5-a431-7f38e44594f0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:03.932420 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1038/2080] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:07:03 2022] DELETE /identity/v3/projects/27435bf6f48543a5be061a248c79eafd => generated 0 bytes in 137 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:07:03.936932 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5412d89a-949b-4d6e-a02b-337b155a1609 service neutron] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:03.947076 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5412d89a-949b-4d6e-a02b-337b155a1609 service neutron] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:04.010327 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deacafdd-1d6b-4ad4-914b-286806e22492 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:04.011220 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deacafdd-1d6b-4ad4-914b-286806e22492 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:04.011220 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deacafdd-1d6b-4ad4-914b-286806e22492 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:04.011479 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deacafdd-1d6b-4ad4-914b-286806e22492 admin admin] PATH_INFO: `/v3/users/26eb43af36b74072943b75c446ebd613` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:04.013165 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deacafdd-1d6b-4ad4-914b-286806e22492 admin admin] RBAC: Authorizing `identity:delete_user(user_id=26eb43af36b74072943b75c446ebd613)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:04.015155 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deacafdd-1d6b-4ad4-914b-286806e22492 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:04.071757 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:07:04.073733 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-deacafdd-1d6b-4ad4-914b-286806e22492 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '26eb43af36b74072943b75c446ebd613'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:04.083224 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-deacafdd-1d6b-4ad4-914b-286806e22492 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '26eb43af36b74072943b75c446ebd613'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:04.103230 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-deacafdd-1d6b-4ad4-914b-286806e22492 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '26eb43af36b74072943b75c446ebd613'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:04.115353 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-deacafdd-1d6b-4ad4-914b-286806e22492 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '26eb43af36b74072943b75c446ebd613'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:04.116324 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-deacafdd-1d6b-4ad4-914b-286806e22492 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '26eb43af36b74072943b75c446ebd613'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:04.124012 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1043/2081] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:07:03 2022] DELETE /identity/v3/users/26eb43af36b74072943b75c446ebd613 => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:07:04.532366 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a6c0125-10a5-49e5-a431-7f38e44594f0 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:04.534916 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-3a6c0125-10a5-49e5-a431-7f38e44594f0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:07:04.551977 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a6c0125-10a5-49e5-a431-7f38e44594f0 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:04.626419 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e893ede-3fe3-4b30-97d2-c15471d599a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:04.627539 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e893ede-3fe3-4b30-97d2-c15471d599a3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:04.628032 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e893ede-3fe3-4b30-97d2-c15471d599a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:04.628590 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e893ede-3fe3-4b30-97d2-c15471d599a3 admin admin] PATH_INFO: `/v3/projects/b92714ece8074d50ac508ddccd193850` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:04.630034 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e893ede-3fe3-4b30-97d2-c15471d599a3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b92714ece8074d50ac508ddccd193850)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:04.632026 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e893ede-3fe3-4b30-97d2-c15471d599a3 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:04.689647 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-9e893ede-3fe3-4b30-97d2-c15471d599a3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b92714ece8074d50ac508ddccd193850', 'request_id': 'req-9e893ede-3fe3-4b30-97d2-c15471d599a3'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:04.697238 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-9e893ede-3fe3-4b30-97d2-c15471d599a3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b92714ece8074d50ac508ddccd193850', 'request_id': 'req-9e893ede-3fe3-4b30-97d2-c15471d599a3'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:04.710667 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-9e893ede-3fe3-4b30-97d2-c15471d599a3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b92714ece8074d50ac508ddccd193850', 'request_id': 'req-9e893ede-3fe3-4b30-97d2-c15471d599a3'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:04.726870 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-9e893ede-3fe3-4b30-97d2-c15471d599a3 admin admin] The token cache is being invalidate because project b92714ece8074d50ac508ddccd193850 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81562) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:07:04.727314 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-9e893ede-3fe3-4b30-97d2-c15471d599a3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:04.727776 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-9e893ede-3fe3-4b30-97d2-c15471d599a3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:04.729612 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1039/2082] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:07:04 2022] DELETE /identity/v3/projects/b92714ece8074d50ac508ddccd193850 => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:07:04.734349 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c87a524-c71c-4b68-a34b-79f3307eb5b5 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:04.734947 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c87a524-c71c-4b68-a34b-79f3307eb5b5 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:04.735068 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c87a524-c71c-4b68-a34b-79f3307eb5b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:04.755816 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-2c87a524-c71c-4b68-a34b-79f3307eb5b5 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:07:04.757158 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-2c87a524-c71c-4b68-a34b-79f3307eb5b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:07:04.820236 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1044/2083] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:07:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 20 09:07:04.828356 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e90d7a7-23d3-4862-b500-8f2bdcb0c8e8 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:04.828648 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e90d7a7-23d3-4862-b500-8f2bdcb0c8e8 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:04.828923 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e90d7a7-23d3-4862-b500-8f2bdcb0c8e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:04.851366 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-4e90d7a7-23d3-4862-b500-8f2bdcb0c8e8 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:07:04.852609 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-4e90d7a7-23d3-4862-b500-8f2bdcb0c8e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:07:04.860856 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1040/2084] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:07:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 20 09:07:04.865118 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c87a524-c71c-4b68-a34b-79f3307eb5b5 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:04.876014 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c87a524-c71c-4b68-a34b-79f3307eb5b5 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:04.881045 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df615d83-c566-48fe-a41b-30868741c4a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:04.881654 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df615d83-c566-48fe-a41b-30868741c4a0 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:04.881928 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df615d83-c566-48fe-a41b-30868741c4a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:04.882201 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df615d83-c566-48fe-a41b-30868741c4a0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:04.882884 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df615d83-c566-48fe-a41b-30868741c4a0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 09:07:04.883184 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df615d83-c566-48fe-a41b-30868741c4a0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:04.884825 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df615d83-c566-48fe-a41b-30868741c4a0 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:04.897745 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1045/2085] 10.222.0.36 () {58 vars in 1153 bytes} [Thu Oct 20 09:07:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 09:07:04.902144 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e90d7a7-23d3-4862-b500-8f2bdcb0c8e8 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:04.912375 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e90d7a7-23d3-4862-b500-8f2bdcb0c8e8 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:04.917071 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3d9176b-9b10-4090-a1f6-8367d151e5be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:04.917674 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d9176b-9b10-4090-a1f6-8367d151e5be admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:04.917953 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d9176b-9b10-4090-a1f6-8367d151e5be admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:04.918221 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d9176b-9b10-4090-a1f6-8367d151e5be admin admin] PATH_INFO: `/v3/projects` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:04.919512 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3d9176b-9b10-4090-a1f6-8367d151e5be admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:04.921493 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3d9176b-9b10-4090-a1f6-8367d151e5be admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:04.950699 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1041/2086] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:07:04 2022] POST /identity/v3/projects => generated 383 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 20 09:07:04.954552 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df615d83-c566-48fe-a41b-30868741c4a0 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:04.964981 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df615d83-c566-48fe-a41b-30868741c4a0 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:05.031069 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70e09ce3-6143-48f9-9c1d-6fbf0310a70d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:05.031674 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e09ce3-6143-48f9-9c1d-6fbf0310a70d admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:05.031946 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e09ce3-6143-48f9-9c1d-6fbf0310a70d admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:05.032213 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e09ce3-6143-48f9-9c1d-6fbf0310a70d admin admin] PATH_INFO: `/v3/users` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:05.032878 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70e09ce3-6143-48f9-9c1d-6fbf0310a70d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:05.034889 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70e09ce3-6143-48f9-9c1d-6fbf0310a70d admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:05.060188 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1046/2087] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:07:04 2022] POST /identity/v3/users => generated 292 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:07:05.061690 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3d9176b-9b10-4090-a1f6-8367d151e5be admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:05.073670 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3d9176b-9b10-4090-a1f6-8367d151e5be admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:05.078989 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b752c0-7ec9-4412-a6f4-6994496a6ac8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:05.079662 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b752c0-7ec9-4412-a6f4-6994496a6ac8 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:05.080001 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b752c0-7ec9-4412-a6f4-6994496a6ac8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:05.080457 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b752c0-7ec9-4412-a6f4-6994496a6ac8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:05.081183 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36b752c0-7ec9-4412-a6f4-6994496a6ac8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:05.082870 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36b752c0-7ec9-4412-a6f4-6994496a6ac8 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:05.094498 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1042/2088] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:07:05 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:07:05.098637 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70e09ce3-6143-48f9-9c1d-6fbf0310a70d admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:05.109704 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70e09ce3-6143-48f9-9c1d-6fbf0310a70d admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:05.114581 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32b9bb22-d3a6-485d-8ccd-e7c1ed51ccc2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:05.115194 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b9bb22-d3a6-485d-8ccd-e7c1ed51ccc2 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:05.115475 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b9bb22-d3a6-485d-8ccd-e7c1ed51ccc2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:05.115752 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b9bb22-d3a6-485d-8ccd-e7c1ed51ccc2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:05.116396 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32b9bb22-d3a6-485d-8ccd-e7c1ed51ccc2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:05.118037 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32b9bb22-d3a6-485d-8ccd-e7c1ed51ccc2 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:05.130188 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1047/2089] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:07:05 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:07:05.134369 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b752c0-7ec9-4412-a6f4-6994496a6ac8 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:05.144987 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b752c0-7ec9-4412-a6f4-6994496a6ac8 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:05.149621 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6991a71c-05ce-4f22-93a9-58745782bb17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:05.150272 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6991a71c-05ce-4f22-93a9-58745782bb17 admin admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:05.150570 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6991a71c-05ce-4f22-93a9-58745782bb17 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:05.150839 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6991a71c-05ce-4f22-93a9-58745782bb17 admin admin] PATH_INFO: `/v3/projects/b81cd68afde444fa9c8d3bda60eecd26/users/a5d1c5f8d517489397826a77105eaec8/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:05.166564 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6991a71c-05ce-4f22-93a9-58745782bb17 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b81cd68afde444fa9c8d3bda60eecd26, user_id=a5d1c5f8d517489397826a77105eaec8, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:05.169383 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6991a71c-05ce-4f22-93a9-58745782bb17 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:05.186574 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1043/2090] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:07:05 2022] PUT /identity/v3/projects/b81cd68afde444fa9c8d3bda60eecd26/users/a5d1c5f8d517489397826a77105eaec8/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:07:05.197556 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0136ab83-f821-4e05-a849-efe78e374df3 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:05.197921 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0136ab83-f821-4e05-a849-efe78e374df3 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:05.198270 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0136ab83-f821-4e05-a849-efe78e374df3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:05.229679 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-0136ab83-f821-4e05-a849-efe78e374df3 None None] MFA Rules not processed for user `a5d1c5f8d517489397826a77105eaec8`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:07:05.230917 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-0136ab83-f821-4e05-a849-efe78e374df3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:07:05.248964 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=93,keystone:INSERT=4,keystone:DELETE=3 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:07:05.311661 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1048/2091] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:07:05 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:07:05.316286 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6991a71c-05ce-4f22-93a9-58745782bb17 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:05.373553 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:07:05.474782 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6991a71c-05ce-4f22-93a9-58745782bb17 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:05.483684 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1acd48c7-c2f0-40b1-b206-2ef7d11b0565 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:05.484314 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1acd48c7-c2f0-40b1-b206-2ef7d11b0565 admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:05.484586 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1acd48c7-c2f0-40b1-b206-2ef7d11b0565 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:05.484851 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1acd48c7-c2f0-40b1-b206-2ef7d11b0565 admin admin] PATH_INFO: `/v3/projects` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:05.485492 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1acd48c7-c2f0-40b1-b206-2ef7d11b0565 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:05.489258 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1acd48c7-c2f0-40b1-b206-2ef7d11b0565 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:05.534402 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1044/2092] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:07:05 2022] POST /identity/v3/projects => generated 383 bytes in 219 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:07:05.538729 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0136ab83-f821-4e05-a849-efe78e374df3 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:05.567117 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0136ab83-f821-4e05-a849-efe78e374df3 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:05.661142 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d75f429-ba56-422f-8323-c57c40576669 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:05.661941 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d75f429-ba56-422f-8323-c57c40576669 admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:05.662529 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d75f429-ba56-422f-8323-c57c40576669 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:05.662951 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d75f429-ba56-422f-8323-c57c40576669 admin admin] PATH_INFO: `/v3/users` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:05.664288 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d75f429-ba56-422f-8323-c57c40576669 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:05.667310 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d75f429-ba56-422f-8323-c57c40576669 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:05.699500 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1049/2093] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:07:05 2022] POST /identity/v3/users => generated 298 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 20 09:07:05.703413 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1acd48c7-c2f0-40b1-b206-2ef7d11b0565 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:05.714177 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1acd48c7-c2f0-40b1-b206-2ef7d11b0565 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:05.722598 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f4288a4-9f34-406a-996d-ad8b3a72c012 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:05.723230 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f4288a4-9f34-406a-996d-ad8b3a72c012 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:05.723750 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f4288a4-9f34-406a-996d-ad8b3a72c012 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:05.723939 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f4288a4-9f34-406a-996d-ad8b3a72c012 admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:05.724583 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f4288a4-9f34-406a-996d-ad8b3a72c012 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:05.726189 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f4288a4-9f34-406a-996d-ad8b3a72c012 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:05.742651 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d75f429-ba56-422f-8323-c57c40576669 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:05.743178 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1045/2094] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:07:05 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 20 09:07:05.757029 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d75f429-ba56-422f-8323-c57c40576669 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:05.761741 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bbb89b5-661d-4259-af07-dc557b3e1679 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:05.762551 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bbb89b5-661d-4259-af07-dc557b3e1679 admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:05.762832 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bbb89b5-661d-4259-af07-dc557b3e1679 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:05.763103 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bbb89b5-661d-4259-af07-dc557b3e1679 admin admin] PATH_INFO: `/v3/projects/1c6ca73bdcee414a86d80c4d579a5b10/users/bb73b34af72a401d9b328e3daa47c063/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:05.779039 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bbb89b5-661d-4259-af07-dc557b3e1679 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1c6ca73bdcee414a86d80c4d579a5b10, user_id=bb73b34af72a401d9b328e3daa47c063, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:05.782148 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bbb89b5-661d-4259-af07-dc557b3e1679 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:05.798236 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1050/2095] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:07:05 2022] PUT /identity/v3/projects/1c6ca73bdcee414a86d80c4d579a5b10/users/bb73b34af72a401d9b328e3daa47c063/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:07:05.802232 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f4288a4-9f34-406a-996d-ad8b3a72c012 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:05.813250 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f4288a4-9f34-406a-996d-ad8b3a72c012 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:05.873181 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c707bffa-9ce7-4870-b36f-e1ce3d53dda5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:05.873887 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c707bffa-9ce7-4870-b36f-e1ce3d53dda5 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:05.874244 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c707bffa-9ce7-4870-b36f-e1ce3d53dda5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:05.874611 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c707bffa-9ce7-4870-b36f-e1ce3d53dda5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:05.875314 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c707bffa-9ce7-4870-b36f-e1ce3d53dda5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:05.876976 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c707bffa-9ce7-4870-b36f-e1ce3d53dda5 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:05.889722 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1046/2096] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:07:05 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:07:05.894123 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bbb89b5-661d-4259-af07-dc557b3e1679 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:05.906619 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bbb89b5-661d-4259-af07-dc557b3e1679 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:05.911464 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cb41b2b-91a1-4869-98cf-412595cc2244 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:05.912125 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb41b2b-91a1-4869-98cf-412595cc2244 admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:05.912403 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb41b2b-91a1-4869-98cf-412595cc2244 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:05.912673 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb41b2b-91a1-4869-98cf-412595cc2244 admin admin] PATH_INFO: `/v3/projects/1c6ca73bdcee414a86d80c4d579a5b10/users/bb73b34af72a401d9b328e3daa47c063/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:05.914615 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cb41b2b-91a1-4869-98cf-412595cc2244 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1c6ca73bdcee414a86d80c4d579a5b10, user_id=bb73b34af72a401d9b328e3daa47c063, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:05.917401 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cb41b2b-91a1-4869-98cf-412595cc2244 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:05.932338 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1051/2097] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:07:05 2022] PUT /identity/v3/projects/1c6ca73bdcee414a86d80c4d579a5b10/users/bb73b34af72a401d9b328e3daa47c063/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:07:05.942257 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78e9491-97d0-46d6-85dd-6869625defa3 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:05.942556 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78e9491-97d0-46d6-85dd-6869625defa3 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:05.942874 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78e9491-97d0-46d6-85dd-6869625defa3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:05.985805 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-a78e9491-97d0-46d6-85dd-6869625defa3 None None] MFA Rules not processed for user `bb73b34af72a401d9b328e3daa47c063`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:07:05.987166 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-a78e9491-97d0-46d6-85dd-6869625defa3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:07:06.062711 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1047/2098] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:07:05 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:07:06.076065 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cb41b2b-91a1-4869-98cf-412595cc2244 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:06.078022 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-8cb41b2b-91a1-4869-98cf-412595cc2244 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:07:06.139712 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cb41b2b-91a1-4869-98cf-412595cc2244 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:06.143392 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07f46d8d-7dd7-478c-a4f0-4fdccfb447e8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:06.144219 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f46d8d-7dd7-478c-a4f0-4fdccfb447e8 service cinder] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:06.144557 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f46d8d-7dd7-478c-a4f0-4fdccfb447e8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:06.144889 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f46d8d-7dd7-478c-a4f0-4fdccfb447e8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:06.155994 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07f46d8d-7dd7-478c-a4f0-4fdccfb447e8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:06.157753 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07f46d8d-7dd7-478c-a4f0-4fdccfb447e8 service cinder] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:06.169746 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1052/2099] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:07:06 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:07:06.284313 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a78e9491-97d0-46d6-85dd-6869625defa3 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:06.294940 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a78e9491-97d0-46d6-85dd-6869625defa3 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:06.300744 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d17d8dc1-e90f-4a8b-a53f-a5cdde3d5812 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:06.301643 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17d8dc1-e90f-4a8b-a53f-a5cdde3d5812 service cinder] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:06.301923 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17d8dc1-e90f-4a8b-a53f-a5cdde3d5812 service cinder] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:06.302195 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17d8dc1-e90f-4a8b-a53f-a5cdde3d5812 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:06.320659 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:07:06.369587 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d17d8dc1-e90f-4a8b-a53f-a5cdde3d5812 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:06.371083 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d17d8dc1-e90f-4a8b-a53f-a5cdde3d5812 service cinder] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:06.382277 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1048/2100] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:07:06 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:07:09.070064 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07f46d8d-7dd7-478c-a4f0-4fdccfb447e8 service cinder] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:09.089842 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:07:09.156507 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07f46d8d-7dd7-478c-a4f0-4fdccfb447e8 service cinder] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:09.162462 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a91cbdd9-e1e2-4ebe-a276-40d995cb9fd1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:09.166513 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a91cbdd9-e1e2-4ebe-a276-40d995cb9fd1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:09.166513 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a91cbdd9-e1e2-4ebe-a276-40d995cb9fd1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:09.166513 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a91cbdd9-e1e2-4ebe-a276-40d995cb9fd1 admin admin] PATH_INFO: `/v3/users/a5d1c5f8d517489397826a77105eaec8` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:09.170224 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a91cbdd9-e1e2-4ebe-a276-40d995cb9fd1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a5d1c5f8d517489397826a77105eaec8)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:09.172410 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a91cbdd9-e1e2-4ebe-a276-40d995cb9fd1 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:09.250853 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-a91cbdd9-e1e2-4ebe-a276-40d995cb9fd1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a5d1c5f8d517489397826a77105eaec8'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:09.259388 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-a91cbdd9-e1e2-4ebe-a276-40d995cb9fd1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a5d1c5f8d517489397826a77105eaec8'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:09.286129 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-a91cbdd9-e1e2-4ebe-a276-40d995cb9fd1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a5d1c5f8d517489397826a77105eaec8'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:09.296270 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-a91cbdd9-e1e2-4ebe-a276-40d995cb9fd1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a5d1c5f8d517489397826a77105eaec8'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:09.299574 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-a91cbdd9-e1e2-4ebe-a276-40d995cb9fd1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a5d1c5f8d517489397826a77105eaec8'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:09.306913 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1053/2101] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:07:09 2022] DELETE /identity/v3/users/a5d1c5f8d517489397826a77105eaec8 => generated 0 bytes in 237 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:07:09.311583 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-385e77db-4419-44c1-bdca-8999d5b1320e None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:09.312106 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-385e77db-4419-44c1-bdca-8999d5b1320e None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:09.312106 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-385e77db-4419-44c1-bdca-8999d5b1320e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:09.332905 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-385e77db-4419-44c1-bdca-8999d5b1320e None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:07:09.334797 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-385e77db-4419-44c1-bdca-8999d5b1320e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:07:09.430306 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1049/2102] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:07:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:07:09.436857 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a91cbdd9-e1e2-4ebe-a276-40d995cb9fd1 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:09.438247 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-a91cbdd9-e1e2-4ebe-a276-40d995cb9fd1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:07:09.507360 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a91cbdd9-e1e2-4ebe-a276-40d995cb9fd1 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:09.511238 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-839acfde-cb1b-43ec-a95f-a46a4601dff0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:09.511838 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-839acfde-cb1b-43ec-a95f-a46a4601dff0 service neutron] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:09.512110 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-839acfde-cb1b-43ec-a95f-a46a4601dff0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:09.512904 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-839acfde-cb1b-43ec-a95f-a46a4601dff0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:09.522193 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-839acfde-cb1b-43ec-a95f-a46a4601dff0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:09.523651 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-839acfde-cb1b-43ec-a95f-a46a4601dff0 service neutron] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:09.535129 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1054/2103] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:07:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:07:09.803874 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-385e77db-4419-44c1-bdca-8999d5b1320e None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:09.805337 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-385e77db-4419-44c1-bdca-8999d5b1320e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:07:09.820029 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-385e77db-4419-44c1-bdca-8999d5b1320e None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:09.826790 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcfb9540-36eb-4988-ad93-fc89feec7f07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:09.827351 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcfb9540-36eb-4988-ad93-fc89feec7f07 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:09.827630 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcfb9540-36eb-4988-ad93-fc89feec7f07 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:09.827900 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcfb9540-36eb-4988-ad93-fc89feec7f07 admin admin] PATH_INFO: `/v3/projects/b81cd68afde444fa9c8d3bda60eecd26` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:09.829001 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcfb9540-36eb-4988-ad93-fc89feec7f07 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b81cd68afde444fa9c8d3bda60eecd26)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:09.830997 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcfb9540-36eb-4988-ad93-fc89feec7f07 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:09.878270 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-bcfb9540-36eb-4988-ad93-fc89feec7f07 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b81cd68afde444fa9c8d3bda60eecd26', 'request_id': 'req-bcfb9540-36eb-4988-ad93-fc89feec7f07'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:09.884600 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-bcfb9540-36eb-4988-ad93-fc89feec7f07 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b81cd68afde444fa9c8d3bda60eecd26', 'request_id': 'req-bcfb9540-36eb-4988-ad93-fc89feec7f07'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:09.890167 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-bcfb9540-36eb-4988-ad93-fc89feec7f07 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b81cd68afde444fa9c8d3bda60eecd26', 'request_id': 'req-bcfb9540-36eb-4988-ad93-fc89feec7f07'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:09.901990 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-bcfb9540-36eb-4988-ad93-fc89feec7f07 admin admin] The token cache is being invalidate because project b81cd68afde444fa9c8d3bda60eecd26 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81562) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:07:09.902278 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-bcfb9540-36eb-4988-ad93-fc89feec7f07 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:09.903479 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-bcfb9540-36eb-4988-ad93-fc89feec7f07 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:09.904942 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1050/2104] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:07:09 2022] DELETE /identity/v3/projects/b81cd68afde444fa9c8d3bda60eecd26 => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:07:09.908928 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-839acfde-cb1b-43ec-a95f-a46a4601dff0 service neutron] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:09.919419 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-839acfde-cb1b-43ec-a95f-a46a4601dff0 service neutron] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:09.951525 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:07:09.978674 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a73abb-6647-404a-9074-b385e89e0c0e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:09.979305 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a73abb-6647-404a-9074-b385e89e0c0e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:09.979586 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a73abb-6647-404a-9074-b385e89e0c0e admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:09.979855 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a73abb-6647-404a-9074-b385e89e0c0e admin admin] PATH_INFO: `/v3/users/bb73b34af72a401d9b328e3daa47c063` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:09.980965 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46a73abb-6647-404a-9074-b385e89e0c0e admin admin] RBAC: Authorizing `identity:delete_user(user_id=bb73b34af72a401d9b328e3daa47c063)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:09.982959 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46a73abb-6647-404a-9074-b385e89e0c0e admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:10.028742 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-46a73abb-6647-404a-9074-b385e89e0c0e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bb73b34af72a401d9b328e3daa47c063'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:10.036699 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-46a73abb-6647-404a-9074-b385e89e0c0e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bb73b34af72a401d9b328e3daa47c063'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:10.050783 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-46a73abb-6647-404a-9074-b385e89e0c0e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bb73b34af72a401d9b328e3daa47c063'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:10.057489 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-46a73abb-6647-404a-9074-b385e89e0c0e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bb73b34af72a401d9b328e3daa47c063'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:10.058148 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-46a73abb-6647-404a-9074-b385e89e0c0e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bb73b34af72a401d9b328e3daa47c063'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:10.064698 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1055/2105] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:07:09 2022] DELETE /identity/v3/users/bb73b34af72a401d9b328e3daa47c063 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:07:10.339613 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcfb9540-36eb-4988-ad93-fc89feec7f07 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:10.342623 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-bcfb9540-36eb-4988-ad93-fc89feec7f07 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:07:10.359299 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcfb9540-36eb-4988-ad93-fc89feec7f07 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:10.405946 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:07:10.424529 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ce144cc-856e-47ce-96ab-046ad1a6de1f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:10.425160 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ce144cc-856e-47ce-96ab-046ad1a6de1f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:10.425464 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ce144cc-856e-47ce-96ab-046ad1a6de1f admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:10.425728 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ce144cc-856e-47ce-96ab-046ad1a6de1f admin admin] PATH_INFO: `/v3/projects/1c6ca73bdcee414a86d80c4d579a5b10` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:10.426845 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ce144cc-856e-47ce-96ab-046ad1a6de1f admin admin] RBAC: Authorizing `identity:delete_project(project_id=1c6ca73bdcee414a86d80c4d579a5b10)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:10.428792 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ce144cc-856e-47ce-96ab-046ad1a6de1f admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:10.479430 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-9ce144cc-856e-47ce-96ab-046ad1a6de1f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1c6ca73bdcee414a86d80c4d579a5b10', 'request_id': 'req-9ce144cc-856e-47ce-96ab-046ad1a6de1f'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:10.487220 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-9ce144cc-856e-47ce-96ab-046ad1a6de1f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1c6ca73bdcee414a86d80c4d579a5b10', 'request_id': 'req-9ce144cc-856e-47ce-96ab-046ad1a6de1f'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:10.494237 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-9ce144cc-856e-47ce-96ab-046ad1a6de1f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1c6ca73bdcee414a86d80c4d579a5b10', 'request_id': 'req-9ce144cc-856e-47ce-96ab-046ad1a6de1f'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:10.508178 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-9ce144cc-856e-47ce-96ab-046ad1a6de1f admin admin] The token cache is being invalidate because project 1c6ca73bdcee414a86d80c4d579a5b10 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81562) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:07:10.508467 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-9ce144cc-856e-47ce-96ab-046ad1a6de1f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:10.509116 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-9ce144cc-856e-47ce-96ab-046ad1a6de1f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:10.510700 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1051/2106] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:07:10 2022] DELETE /identity/v3/projects/1c6ca73bdcee414a86d80c4d579a5b10 => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:07:10.516203 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f880af4f-232f-46cc-87ec-efefe22eb4c2 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:10.516666 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f880af4f-232f-46cc-87ec-efefe22eb4c2 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:10.516761 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f880af4f-232f-46cc-87ec-efefe22eb4c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:10.536523 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-f880af4f-232f-46cc-87ec-efefe22eb4c2 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:07:10.537725 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-f880af4f-232f-46cc-87ec-efefe22eb4c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:07:10.599872 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1056/2107] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:07:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:07:10.607940 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cfdd647-8cc5-4803-bf2d-9d3276cdd26b None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:10.608221 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cfdd647-8cc5-4803-bf2d-9d3276cdd26b None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:10.608490 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cfdd647-8cc5-4803-bf2d-9d3276cdd26b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:10.628543 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-9cfdd647-8cc5-4803-bf2d-9d3276cdd26b None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:07:10.629800 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-9cfdd647-8cc5-4803-bf2d-9d3276cdd26b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:07:10.639049 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1052/2108] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:07:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:07:10.643388 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f880af4f-232f-46cc-87ec-efefe22eb4c2 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:10.653833 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f880af4f-232f-46cc-87ec-efefe22eb4c2 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:10.658489 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9fb346a-90a0-497d-93b7-bc83176718e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:10.659023 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9fb346a-90a0-497d-93b7-bc83176718e4 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:10.659297 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9fb346a-90a0-497d-93b7-bc83176718e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:10.659565 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9fb346a-90a0-497d-93b7-bc83176718e4 admin admin] PATH_INFO: `/v3/domains` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:10.660202 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9fb346a-90a0-497d-93b7-bc83176718e4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 09:07:10.660501 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9fb346a-90a0-497d-93b7-bc83176718e4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:10.662087 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9fb346a-90a0-497d-93b7-bc83176718e4 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:10.674137 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1057/2109] 10.222.0.36 () {58 vars in 1153 bytes} [Thu Oct 20 09:07:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 09:07:10.678331 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cfdd647-8cc5-4803-bf2d-9d3276cdd26b None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:10.689350 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cfdd647-8cc5-4803-bf2d-9d3276cdd26b None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:10.693866 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc84536c-e8e7-4fab-870a-a30c26255af2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:10.694451 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc84536c-e8e7-4fab-870a-a30c26255af2 admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:10.694759 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc84536c-e8e7-4fab-870a-a30c26255af2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:10.695054 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc84536c-e8e7-4fab-870a-a30c26255af2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:10.695675 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc84536c-e8e7-4fab-870a-a30c26255af2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:10.697555 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc84536c-e8e7-4fab-870a-a30c26255af2 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:10.725100 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1053/2110] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:07:10 2022] POST /identity/v3/projects => generated 381 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 20 09:07:10.729230 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9fb346a-90a0-497d-93b7-bc83176718e4 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:10.739858 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9fb346a-90a0-497d-93b7-bc83176718e4 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:10.778659 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:07:10.796654 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5d90f68-34ca-4f83-8fd2-0742d640cb0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:10.797246 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d90f68-34ca-4f83-8fd2-0742d640cb0b admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:10.797521 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d90f68-34ca-4f83-8fd2-0742d640cb0b admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:10.797788 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d90f68-34ca-4f83-8fd2-0742d640cb0b admin admin] PATH_INFO: `/v3/users` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:10.798410 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5d90f68-34ca-4f83-8fd2-0742d640cb0b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:10.800377 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5d90f68-34ca-4f83-8fd2-0742d640cb0b admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:10.821469 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1058/2111] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:07:10 2022] POST /identity/v3/users => generated 291 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:07:10.826145 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc84536c-e8e7-4fab-870a-a30c26255af2 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:10.836153 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc84536c-e8e7-4fab-870a-a30c26255af2 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:10.841579 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59ab4450-7d95-4c1d-bf47-48f93b7dd34c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:10.842153 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59ab4450-7d95-4c1d-bf47-48f93b7dd34c admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:10.842422 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59ab4450-7d95-4c1d-bf47-48f93b7dd34c admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:10.842716 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59ab4450-7d95-4c1d-bf47-48f93b7dd34c admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:10.843346 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59ab4450-7d95-4c1d-bf47-48f93b7dd34c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:10.844919 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59ab4450-7d95-4c1d-bf47-48f93b7dd34c admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:10.854674 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1054/2112] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:07:10 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:07:10.858695 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5d90f68-34ca-4f83-8fd2-0742d640cb0b admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:10.868402 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5d90f68-34ca-4f83-8fd2-0742d640cb0b admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:10.873088 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-522e8b6a-d422-4a27-b759-a84f1be8d2f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:10.873664 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-522e8b6a-d422-4a27-b759-a84f1be8d2f2 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:10.873938 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-522e8b6a-d422-4a27-b759-a84f1be8d2f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:10.874228 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-522e8b6a-d422-4a27-b759-a84f1be8d2f2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:10.874892 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-522e8b6a-d422-4a27-b759-a84f1be8d2f2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:10.876568 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-522e8b6a-d422-4a27-b759-a84f1be8d2f2 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:10.890640 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1059/2113] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:07:10 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:07:10.894800 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59ab4450-7d95-4c1d-bf47-48f93b7dd34c admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:10.905224 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59ab4450-7d95-4c1d-bf47-48f93b7dd34c admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:10.909830 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5556712d-dcc6-470b-83a9-f8b1259186f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:10.910405 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5556712d-dcc6-470b-83a9-f8b1259186f8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:10.910714 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5556712d-dcc6-470b-83a9-f8b1259186f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:10.910987 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5556712d-dcc6-470b-83a9-f8b1259186f8 admin admin] PATH_INFO: `/v3/projects/b97f91d8fd9a4d80afb2250ed448f29f/users/666be326f50f4074b8a49130de80fdd7/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:10.925663 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5556712d-dcc6-470b-83a9-f8b1259186f8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b97f91d8fd9a4d80afb2250ed448f29f, user_id=666be326f50f4074b8a49130de80fdd7, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:10.928359 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5556712d-dcc6-470b-83a9-f8b1259186f8 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:10.942325 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1055/2114] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:07:10 2022] PUT /identity/v3/projects/b97f91d8fd9a4d80afb2250ed448f29f/users/666be326f50f4074b8a49130de80fdd7/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:07:11.009567 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142ed33c-a3a4-43f7-897b-0ca76e9f677d None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:11.009854 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142ed33c-a3a4-43f7-897b-0ca76e9f677d None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:11.010271 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142ed33c-a3a4-43f7-897b-0ca76e9f677d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:11.039065 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-142ed33c-a3a4-43f7-897b-0ca76e9f677d None None] MFA Rules not processed for user `666be326f50f4074b8a49130de80fdd7`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:07:11.040264 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-142ed33c-a3a4-43f7-897b-0ca76e9f677d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:07:11.120864 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1060/2115] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:07:11 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:07:11.125679 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5556712d-dcc6-470b-83a9-f8b1259186f8 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:11.199002 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5556712d-dcc6-470b-83a9-f8b1259186f8 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:11.204613 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f335bfc9-5277-4fa4-bc39-3b0ce68f88ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:11.205322 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f335bfc9-5277-4fa4-bc39-3b0ce68f88ff admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:11.205617 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f335bfc9-5277-4fa4-bc39-3b0ce68f88ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:11.205882 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f335bfc9-5277-4fa4-bc39-3b0ce68f88ff admin admin] PATH_INFO: `/v3/projects` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:11.206684 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f335bfc9-5277-4fa4-bc39-3b0ce68f88ff admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:11.208672 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f335bfc9-5277-4fa4-bc39-3b0ce68f88ff admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:11.229765 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:07:11.248160 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1056/2116] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:07:11 2022] POST /identity/v3/projects => generated 379 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 20 09:07:11.249270 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-142ed33c-a3a4-43f7-897b-0ca76e9f677d None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:11.260052 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-142ed33c-a3a4-43f7-897b-0ca76e9f677d None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:11.316007 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a978c4d1-345c-4039-93ac-51b1b32c3bd5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:11.316609 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a978c4d1-345c-4039-93ac-51b1b32c3bd5 admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:11.316884 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a978c4d1-345c-4039-93ac-51b1b32c3bd5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:11.317153 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a978c4d1-345c-4039-93ac-51b1b32c3bd5 admin admin] PATH_INFO: `/v3/users` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:11.317782 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a978c4d1-345c-4039-93ac-51b1b32c3bd5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:11.319796 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a978c4d1-345c-4039-93ac-51b1b32c3bd5 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:11.341832 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1061/2117] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:07:11 2022] POST /identity/v3/users => generated 296 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 20 09:07:11.345873 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f335bfc9-5277-4fa4-bc39-3b0ce68f88ff admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:11.356163 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f335bfc9-5277-4fa4-bc39-3b0ce68f88ff admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:11.360757 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd26f509-b38b-40d2-958c-28f87ec9fe48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:11.361350 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd26f509-b38b-40d2-958c-28f87ec9fe48 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:11.361628 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd26f509-b38b-40d2-958c-28f87ec9fe48 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:11.361900 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd26f509-b38b-40d2-958c-28f87ec9fe48 admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:11.362571 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd26f509-b38b-40d2-958c-28f87ec9fe48 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:11.364194 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd26f509-b38b-40d2-958c-28f87ec9fe48 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:11.375547 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1057/2118] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:07:11 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:07:11.379815 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a978c4d1-345c-4039-93ac-51b1b32c3bd5 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:11.391326 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a978c4d1-345c-4039-93ac-51b1b32c3bd5 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:11.395847 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c58d9a01-cd96-481e-82e7-676b55bfb39d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:11.396487 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c58d9a01-cd96-481e-82e7-676b55bfb39d admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:11.396765 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c58d9a01-cd96-481e-82e7-676b55bfb39d admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:11.397030 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c58d9a01-cd96-481e-82e7-676b55bfb39d admin admin] PATH_INFO: `/v3/projects/1c67825a6e4342b2855cbfebc04aa365/users/98d0795a75194f60811bb93628a5739a/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:11.411401 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c58d9a01-cd96-481e-82e7-676b55bfb39d admin admin] RBAC: Authorizing `identity:create_grant(project_id=1c67825a6e4342b2855cbfebc04aa365, user_id=98d0795a75194f60811bb93628a5739a, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:11.414050 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c58d9a01-cd96-481e-82e7-676b55bfb39d admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:11.426908 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1062/2119] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:07:11 2022] PUT /identity/v3/projects/1c67825a6e4342b2855cbfebc04aa365/users/98d0795a75194f60811bb93628a5739a/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:07:11.430813 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd26f509-b38b-40d2-958c-28f87ec9fe48 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:11.441264 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd26f509-b38b-40d2-958c-28f87ec9fe48 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:11.502808 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e974ba2-4a2c-4084-b6bd-d26751330a0e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:11.503406 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e974ba2-4a2c-4084-b6bd-d26751330a0e admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:11.503678 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e974ba2-4a2c-4084-b6bd-d26751330a0e admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:11.503944 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e974ba2-4a2c-4084-b6bd-d26751330a0e admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:11.504581 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e974ba2-4a2c-4084-b6bd-d26751330a0e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:11.506156 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e974ba2-4a2c-4084-b6bd-d26751330a0e admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:11.516641 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1058/2120] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:07:11 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:07:11.520652 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c58d9a01-cd96-481e-82e7-676b55bfb39d admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:11.531317 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c58d9a01-cd96-481e-82e7-676b55bfb39d admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:11.536642 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-155b9d0f-fda6-4ee0-8f02-f8664b79b4f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:11.537289 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-155b9d0f-fda6-4ee0-8f02-f8664b79b4f9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:11.537616 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-155b9d0f-fda6-4ee0-8f02-f8664b79b4f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:11.537887 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-155b9d0f-fda6-4ee0-8f02-f8664b79b4f9 admin admin] PATH_INFO: `/v3/projects/1c67825a6e4342b2855cbfebc04aa365/users/98d0795a75194f60811bb93628a5739a/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:11.539878 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-155b9d0f-fda6-4ee0-8f02-f8664b79b4f9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1c67825a6e4342b2855cbfebc04aa365, user_id=98d0795a75194f60811bb93628a5739a, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:11.542612 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-155b9d0f-fda6-4ee0-8f02-f8664b79b4f9 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:11.553992 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1063/2121] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:07:11 2022] PUT /identity/v3/projects/1c67825a6e4342b2855cbfebc04aa365/users/98d0795a75194f60811bb93628a5739a/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:07:11.564268 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9965d924-c8c5-466f-a861-293de7b2af96 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:11.564557 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9965d924-c8c5-466f-a861-293de7b2af96 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:11.564834 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9965d924-c8c5-466f-a861-293de7b2af96 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:11.594404 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-9965d924-c8c5-466f-a861-293de7b2af96 None None] MFA Rules not processed for user `98d0795a75194f60811bb93628a5739a`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:07:11.595652 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-9965d924-c8c5-466f-a861-293de7b2af96 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:07:11.665086 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1059/2122] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:07:11 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:07:11.676902 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-155b9d0f-fda6-4ee0-8f02-f8664b79b4f9 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:11.679989 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-155b9d0f-fda6-4ee0-8f02-f8664b79b4f9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:07:11.685355 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:07:11.741400 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-155b9d0f-fda6-4ee0-8f02-f8664b79b4f9 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:11.745179 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb855e7-19d9-407b-a998-aab0243ecf36 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:11.745886 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb855e7-19d9-407b-a998-aab0243ecf36 service cinder] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:11.746233 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb855e7-19d9-407b-a998-aab0243ecf36 service cinder] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:11.746650 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb855e7-19d9-407b-a998-aab0243ecf36 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:11.755995 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eb855e7-19d9-407b-a998-aab0243ecf36 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:11.757420 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eb855e7-19d9-407b-a998-aab0243ecf36 service cinder] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:11.767515 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1064/2123] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:07:11 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:07:11.842840 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9965d924-c8c5-466f-a861-293de7b2af96 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:11.852478 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9965d924-c8c5-466f-a861-293de7b2af96 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:11.856685 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d57c8e41-be5c-4a42-9a6a-c0ec40e7c184 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:11.857279 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d57c8e41-be5c-4a42-9a6a-c0ec40e7c184 service cinder] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:11.857558 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d57c8e41-be5c-4a42-9a6a-c0ec40e7c184 service cinder] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:11.857826 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d57c8e41-be5c-4a42-9a6a-c0ec40e7c184 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:11.912580 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d57c8e41-be5c-4a42-9a6a-c0ec40e7c184 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:11.913992 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d57c8e41-be5c-4a42-9a6a-c0ec40e7c184 service cinder] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:11.923459 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1060/2124] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:07:11 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 81 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:07:12.185395 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb855e7-19d9-407b-a998-aab0243ecf36 service cinder] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:12.247177 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb855e7-19d9-407b-a998-aab0243ecf36 service cinder] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:12.251620 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d94ccc77-f3a0-4090-9535-bcb4a5a722a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:12.252233 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94ccc77-f3a0-4090-9535-bcb4a5a722a3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:12.252503 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94ccc77-f3a0-4090-9535-bcb4a5a722a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:12.252771 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94ccc77-f3a0-4090-9535-bcb4a5a722a3 admin admin] PATH_INFO: `/v3/users/666be326f50f4074b8a49130de80fdd7` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:12.253858 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d94ccc77-f3a0-4090-9535-bcb4a5a722a3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=666be326f50f4074b8a49130de80fdd7)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:12.255797 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d94ccc77-f3a0-4090-9535-bcb4a5a722a3 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:12.305809 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-d94ccc77-f3a0-4090-9535-bcb4a5a722a3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '666be326f50f4074b8a49130de80fdd7'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:12.319946 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-d94ccc77-f3a0-4090-9535-bcb4a5a722a3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '666be326f50f4074b8a49130de80fdd7'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:12.340266 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-d94ccc77-f3a0-4090-9535-bcb4a5a722a3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '666be326f50f4074b8a49130de80fdd7'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:12.348519 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-d94ccc77-f3a0-4090-9535-bcb4a5a722a3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '666be326f50f4074b8a49130de80fdd7'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:12.349441 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-d94ccc77-f3a0-4090-9535-bcb4a5a722a3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '666be326f50f4074b8a49130de80fdd7'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:12.355667 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1065/2125] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:07:12 2022] DELETE /identity/v3/users/666be326f50f4074b8a49130de80fdd7 => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:07:12.360669 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-951623ca-f0cf-40a0-b7a5-c433d38646d3 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:12.361132 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-951623ca-f0cf-40a0-b7a5-c433d38646d3 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:12.361243 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-951623ca-f0cf-40a0-b7a5-c433d38646d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:12.368459 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:07:12.383315 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-951623ca-f0cf-40a0-b7a5-c433d38646d3 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:07:12.384519 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-951623ca-f0cf-40a0-b7a5-c433d38646d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:07:12.444075 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1061/2126] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:07:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:07:12.454178 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d94ccc77-f3a0-4090-9535-bcb4a5a722a3 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:12.455594 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-d94ccc77-f3a0-4090-9535-bcb4a5a722a3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:07:12.485268 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:07:12.519226 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d94ccc77-f3a0-4090-9535-bcb4a5a722a3 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:12.523588 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-856027be-aede-49fe-9ea8-a2a0d14b6402 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:12.524256 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-856027be-aede-49fe-9ea8-a2a0d14b6402 service neutron] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:12.524530 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-856027be-aede-49fe-9ea8-a2a0d14b6402 service neutron] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:12.524822 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-856027be-aede-49fe-9ea8-a2a0d14b6402 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:12.535107 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-856027be-aede-49fe-9ea8-a2a0d14b6402 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:12.536635 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-856027be-aede-49fe-9ea8-a2a0d14b6402 service neutron] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:12.546389 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1066/2127] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:07:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:07:12.891659 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-951623ca-f0cf-40a0-b7a5-c433d38646d3 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:12.893122 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-951623ca-f0cf-40a0-b7a5-c433d38646d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:07:12.917452 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-951623ca-f0cf-40a0-b7a5-c433d38646d3 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:12.923663 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99c99f2e-23e0-4e5a-b58c-a961fe0e0932 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:12.925017 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c99f2e-23e0-4e5a-b58c-a961fe0e0932 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:12.925297 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c99f2e-23e0-4e5a-b58c-a961fe0e0932 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:12.926351 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c99f2e-23e0-4e5a-b58c-a961fe0e0932 admin admin] PATH_INFO: `/v3/projects/b97f91d8fd9a4d80afb2250ed448f29f` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:12.927608 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99c99f2e-23e0-4e5a-b58c-a961fe0e0932 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b97f91d8fd9a4d80afb2250ed448f29f)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:12.929535 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99c99f2e-23e0-4e5a-b58c-a961fe0e0932 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:12.991171 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-99c99f2e-23e0-4e5a-b58c-a961fe0e0932 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b97f91d8fd9a4d80afb2250ed448f29f', 'request_id': 'req-99c99f2e-23e0-4e5a-b58c-a961fe0e0932'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:13.000325 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-99c99f2e-23e0-4e5a-b58c-a961fe0e0932 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b97f91d8fd9a4d80afb2250ed448f29f', 'request_id': 'req-99c99f2e-23e0-4e5a-b58c-a961fe0e0932'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:13.007821 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-99c99f2e-23e0-4e5a-b58c-a961fe0e0932 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b97f91d8fd9a4d80afb2250ed448f29f', 'request_id': 'req-99c99f2e-23e0-4e5a-b58c-a961fe0e0932'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:13.023762 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-99c99f2e-23e0-4e5a-b58c-a961fe0e0932 admin admin] The token cache is being invalidate because project b97f91d8fd9a4d80afb2250ed448f29f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81562) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:07:13.024079 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-99c99f2e-23e0-4e5a-b58c-a961fe0e0932 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:13.024822 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-99c99f2e-23e0-4e5a-b58c-a961fe0e0932 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:13.026505 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1062/2128] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:07:12 2022] DELETE /identity/v3/projects/b97f91d8fd9a4d80afb2250ed448f29f => generated 0 bytes in 135 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:07:13.039053 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-856027be-aede-49fe-9ea8-a2a0d14b6402 service neutron] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:13.049142 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-856027be-aede-49fe-9ea8-a2a0d14b6402 service neutron] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:13.107210 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-088a5680-41a7-4e58-a41a-798a8805f23a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:13.107835 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-088a5680-41a7-4e58-a41a-798a8805f23a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:13.108106 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-088a5680-41a7-4e58-a41a-798a8805f23a admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:13.108372 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-088a5680-41a7-4e58-a41a-798a8805f23a admin admin] PATH_INFO: `/v3/users/98d0795a75194f60811bb93628a5739a` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:13.109469 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-088a5680-41a7-4e58-a41a-798a8805f23a admin admin] RBAC: Authorizing `identity:delete_user(user_id=98d0795a75194f60811bb93628a5739a)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:13.111450 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-088a5680-41a7-4e58-a41a-798a8805f23a admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:13.159429 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-088a5680-41a7-4e58-a41a-798a8805f23a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '98d0795a75194f60811bb93628a5739a'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:13.166778 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-088a5680-41a7-4e58-a41a-798a8805f23a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '98d0795a75194f60811bb93628a5739a'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:13.181351 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-088a5680-41a7-4e58-a41a-798a8805f23a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '98d0795a75194f60811bb93628a5739a'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:13.189753 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99c99f2e-23e0-4e5a-b58c-a961fe0e0932 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:13.191246 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-99c99f2e-23e0-4e5a-b58c-a961fe0e0932 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:07:13.191504 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-088a5680-41a7-4e58-a41a-798a8805f23a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '98d0795a75194f60811bb93628a5739a'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:13.192196 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-088a5680-41a7-4e58-a41a-798a8805f23a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '98d0795a75194f60811bb93628a5739a'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:13.199761 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1067/2129] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:07:13 2022] DELETE /identity/v3/users/98d0795a75194f60811bb93628a5739a => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:07:13.282116 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99c99f2e-23e0-4e5a-b58c-a961fe0e0932 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:13.309945 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:07:13.374294 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d69b6ba-a921-490a-aeef-b3639cd9523c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a94e868115ec49918eda10724263930c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:13.382316 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d69b6ba-a921-490a-aeef-b3639cd9523c service nova] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:13.383892 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d69b6ba-a921-490a-aeef-b3639cd9523c service nova] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:13.384331 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d69b6ba-a921-490a-aeef-b3639cd9523c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:13.387081 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-1d69b6ba-a921-490a-aeef-b3639cd9523c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:07:13.457482 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d69b6ba-a921-490a-aeef-b3639cd9523c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:13.459055 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d69b6ba-a921-490a-aeef-b3639cd9523c service nova] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:13.469577 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1063/2130] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:07:13 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 280 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 20 09:07:13.530077 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-088a5680-41a7-4e58-a41a-798a8805f23a admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:13.532259 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-088a5680-41a7-4e58-a41a-798a8805f23a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:07:13.592971 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:07:13.617604 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-088a5680-41a7-4e58-a41a-798a8805f23a admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:13.624180 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-469417d3-3d14-4fb7-8c73-8d154d9c16cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:13.624932 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-469417d3-3d14-4fb7-8c73-8d154d9c16cd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:13.625218 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-469417d3-3d14-4fb7-8c73-8d154d9c16cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:13.625486 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-469417d3-3d14-4fb7-8c73-8d154d9c16cd admin admin] PATH_INFO: `/v3/projects/1c67825a6e4342b2855cbfebc04aa365` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:13.629084 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-469417d3-3d14-4fb7-8c73-8d154d9c16cd admin admin] RBAC: Authorizing `identity:delete_project(project_id=1c67825a6e4342b2855cbfebc04aa365)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:13.631078 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-469417d3-3d14-4fb7-8c73-8d154d9c16cd admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:13.690590 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-469417d3-3d14-4fb7-8c73-8d154d9c16cd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1c67825a6e4342b2855cbfebc04aa365', 'request_id': 'req-469417d3-3d14-4fb7-8c73-8d154d9c16cd'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:13.697831 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-469417d3-3d14-4fb7-8c73-8d154d9c16cd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1c67825a6e4342b2855cbfebc04aa365', 'request_id': 'req-469417d3-3d14-4fb7-8c73-8d154d9c16cd'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:13.711780 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-469417d3-3d14-4fb7-8c73-8d154d9c16cd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1c67825a6e4342b2855cbfebc04aa365', 'request_id': 'req-469417d3-3d14-4fb7-8c73-8d154d9c16cd'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:13.728278 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-469417d3-3d14-4fb7-8c73-8d154d9c16cd admin admin] The token cache is being invalidate because project 1c67825a6e4342b2855cbfebc04aa365 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81561) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:07:13.728599 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-469417d3-3d14-4fb7-8c73-8d154d9c16cd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:13.729895 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-469417d3-3d14-4fb7-8c73-8d154d9c16cd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:13.731758 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1068/2131] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:07:13 2022] DELETE /identity/v3/projects/1c67825a6e4342b2855cbfebc04aa365 => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:07:13.738666 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b01619-ccc6-484d-9805-bb7fc8d2ce35 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:13.740382 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b01619-ccc6-484d-9805-bb7fc8d2ce35 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:13.740738 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b01619-ccc6-484d-9805-bb7fc8d2ce35 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:13.785462 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-96b01619-ccc6-484d-9805-bb7fc8d2ce35 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:07:13.786892 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-96b01619-ccc6-484d-9805-bb7fc8d2ce35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:07:13.861427 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1064/2132] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:07:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:07:13.869988 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fda36bc-7745-4cf0-ad23-bce9d3f8580f None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:13.870271 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fda36bc-7745-4cf0-ad23-bce9d3f8580f None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:13.870597 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fda36bc-7745-4cf0-ad23-bce9d3f8580f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:13.893123 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-6fda36bc-7745-4cf0-ad23-bce9d3f8580f None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:07:13.894412 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-6fda36bc-7745-4cf0-ad23-bce9d3f8580f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:07:13.903262 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1069/2133] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:07:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:07:13.907469 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b01619-ccc6-484d-9805-bb7fc8d2ce35 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:13.917629 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b01619-ccc6-484d-9805-bb7fc8d2ce35 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:13.922367 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54fceffc-ac9a-4689-b552-881f8e2d12dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:13.923016 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54fceffc-ac9a-4689-b552-881f8e2d12dc admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:13.923293 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54fceffc-ac9a-4689-b552-881f8e2d12dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:13.923561 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54fceffc-ac9a-4689-b552-881f8e2d12dc admin admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:13.924202 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54fceffc-ac9a-4689-b552-881f8e2d12dc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 09:07:13.924501 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54fceffc-ac9a-4689-b552-881f8e2d12dc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:13.926095 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54fceffc-ac9a-4689-b552-881f8e2d12dc admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:13.937964 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1065/2134] 10.222.0.36 () {58 vars in 1153 bytes} [Thu Oct 20 09:07:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 09:07:13.942073 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fda36bc-7745-4cf0-ad23-bce9d3f8580f None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:13.952039 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fda36bc-7745-4cf0-ad23-bce9d3f8580f None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:13.956684 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ef715d6-e17f-46ae-89c7-407e2a867ea8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:13.957397 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef715d6-e17f-46ae-89c7-407e2a867ea8 admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:13.957731 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef715d6-e17f-46ae-89c7-407e2a867ea8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:13.957971 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef715d6-e17f-46ae-89c7-407e2a867ea8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:13.958626 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ef715d6-e17f-46ae-89c7-407e2a867ea8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:13.960526 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ef715d6-e17f-46ae-89c7-407e2a867ea8 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:13.991871 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1070/2135] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:07:13 2022] POST /identity/v3/projects => generated 385 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:07:13.992979 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54fceffc-ac9a-4689-b552-881f8e2d12dc admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:14.002852 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54fceffc-ac9a-4689-b552-881f8e2d12dc admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:14.061744 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:07:14.076488 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb2a7f1-ac93-49d1-800e-ea296b597cbb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:14.077324 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb2a7f1-ac93-49d1-800e-ea296b597cbb admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:14.077544 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb2a7f1-ac93-49d1-800e-ea296b597cbb admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:14.077882 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb2a7f1-ac93-49d1-800e-ea296b597cbb admin admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:14.078750 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdb2a7f1-ac93-49d1-800e-ea296b597cbb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:14.081648 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdb2a7f1-ac93-49d1-800e-ea296b597cbb admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:14.113727 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1066/2136] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:07:13 2022] POST /identity/v3/users => generated 293 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 20 09:07:14.114700 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ef715d6-e17f-46ae-89c7-407e2a867ea8 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:14.127299 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ef715d6-e17f-46ae-89c7-407e2a867ea8 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:14.132714 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93a28ffd-0567-499b-9ff6-420bfd3d3702 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:14.133547 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a28ffd-0567-499b-9ff6-420bfd3d3702 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:14.133982 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a28ffd-0567-499b-9ff6-420bfd3d3702 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:14.134423 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a28ffd-0567-499b-9ff6-420bfd3d3702 admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:14.135327 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93a28ffd-0567-499b-9ff6-420bfd3d3702 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:14.137148 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93a28ffd-0567-499b-9ff6-420bfd3d3702 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:14.148535 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1071/2137] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:07:14 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:07:14.153775 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb2a7f1-ac93-49d1-800e-ea296b597cbb admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:14.164287 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb2a7f1-ac93-49d1-800e-ea296b597cbb admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:14.172761 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99dd8e82-acfe-48f5-8693-b4f4737f1363 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:14.174599 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99dd8e82-acfe-48f5-8693-b4f4737f1363 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:14.175729 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99dd8e82-acfe-48f5-8693-b4f4737f1363 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:14.176042 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99dd8e82-acfe-48f5-8693-b4f4737f1363 admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:14.176736 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99dd8e82-acfe-48f5-8693-b4f4737f1363 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:14.178712 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99dd8e82-acfe-48f5-8693-b4f4737f1363 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:14.198844 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93a28ffd-0567-499b-9ff6-420bfd3d3702 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:14.199724 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1067/2138] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:07:14 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:07:14.212883 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93a28ffd-0567-499b-9ff6-420bfd3d3702 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:14.228247 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf0cb1c-60b0-480d-b519-f9d304abb00e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:14.230629 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf0cb1c-60b0-480d-b519-f9d304abb00e admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:14.232119 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf0cb1c-60b0-480d-b519-f9d304abb00e admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:14.232489 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf0cb1c-60b0-480d-b519-f9d304abb00e admin admin] PATH_INFO: `/v3/projects/03478f9b92284bd7ac1f16ddeffb87f1/users/38ffa549cacd4af29c9331077a5eca58/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:14.257658 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcf0cb1c-60b0-480d-b519-f9d304abb00e admin admin] RBAC: Authorizing `identity:create_grant(project_id=03478f9b92284bd7ac1f16ddeffb87f1, user_id=38ffa549cacd4af29c9331077a5eca58, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:14.261645 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcf0cb1c-60b0-480d-b519-f9d304abb00e admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:14.278330 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1072/2139] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:07:14 2022] PUT /identity/v3/projects/03478f9b92284bd7ac1f16ddeffb87f1/users/38ffa549cacd4af29c9331077a5eca58/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 80 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:07:14.289401 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0614e5bd-ca1b-48b1-bd56-1f0368d6df42 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:14.289700 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0614e5bd-ca1b-48b1-bd56-1f0368d6df42 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:14.289968 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0614e5bd-ca1b-48b1-bd56-1f0368d6df42 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:14.322493 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-0614e5bd-ca1b-48b1-bd56-1f0368d6df42 None None] MFA Rules not processed for user `38ffa549cacd4af29c9331077a5eca58`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:07:14.324000 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-0614e5bd-ca1b-48b1-bd56-1f0368d6df42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:07:14.404409 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1068/2140] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:07:14 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:07:14.415675 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf0cb1c-60b0-480d-b519-f9d304abb00e admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:14.417462 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-bcf0cb1c-60b0-480d-b519-f9d304abb00e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:07:14.480064 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf0cb1c-60b0-480d-b519-f9d304abb00e admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:14.486395 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fbf5646-b13f-4529-a805-3c7228219ba1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:14.487034 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fbf5646-b13f-4529-a805-3c7228219ba1 service cinder] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:14.487310 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fbf5646-b13f-4529-a805-3c7228219ba1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:14.487577 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fbf5646-b13f-4529-a805-3c7228219ba1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:14.496352 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fbf5646-b13f-4529-a805-3c7228219ba1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:14.498334 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fbf5646-b13f-4529-a805-3c7228219ba1 service cinder] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:14.507531 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1073/2141] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:07:14 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:07:21.699638 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8941968-c225-4845-815c-c22e50ce904a None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:21.700517 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8941968-c225-4845-815c-c22e50ce904a None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:21.700789 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8941968-c225-4845-815c-c22e50ce904a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:21.720369 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-a8941968-c225-4845-815c-c22e50ce904a None None] MFA Rules not processed for user `f4b0e9187ce74f7888e9e7094be5c824`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:07:21.721566 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-a8941968-c225-4845-815c-c22e50ce904a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:07:21.778452 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1069/2142] 10.222.0.36 () {58 vars in 967 bytes} [Thu Oct 20 09:07:21 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:07:21.791556 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fbf5646-b13f-4529-a805-3c7228219ba1 service cinder] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:21.793010 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-9fbf5646-b13f-4529-a805-3c7228219ba1 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:07:21.811850 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:07:21.864035 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fbf5646-b13f-4529-a805-3c7228219ba1 service cinder] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:21.868982 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83004e9c-9e20-4a7e-aa5d-240c0f7c0ff0 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4d558918cd8b4ebea4fe8765297304db', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:21.869580 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83004e9c-9e20-4a7e-aa5d-240c0f7c0ff0 service swift] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:21.869857 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83004e9c-9e20-4a7e-aa5d-240c0f7c0ff0 service swift] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:21.870126 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83004e9c-9e20-4a7e-aa5d-240c0f7c0ff0 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:21.878149 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83004e9c-9e20-4a7e-aa5d-240c0f7c0ff0 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:21.879775 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83004e9c-9e20-4a7e-aa5d-240c0f7c0ff0 service swift] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:21.888019 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1074/2143] 10.222.0.36 () {60 vars in 1384 bytes} [Thu Oct 20 09:07:21 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 20 09:07:21.978871 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8941968-c225-4845-815c-c22e50ce904a None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:22.046607 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8941968-c225-4845-815c-c22e50ce904a None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:22.052015 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c571f5d7-6f18-41f5-a33d-1e82e11bff52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:22.052646 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c571f5d7-6f18-41f5-a33d-1e82e11bff52 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:22.053062 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c571f5d7-6f18-41f5-a33d-1e82e11bff52 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:22.053180 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c571f5d7-6f18-41f5-a33d-1e82e11bff52 admin admin] PATH_INFO: `/v3/users/38ffa549cacd4af29c9331077a5eca58` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:22.054331 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c571f5d7-6f18-41f5-a33d-1e82e11bff52 admin admin] RBAC: Authorizing `identity:delete_user(user_id=38ffa549cacd4af29c9331077a5eca58)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:22.056281 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c571f5d7-6f18-41f5-a33d-1e82e11bff52 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:22.059605 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:07:22.109927 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-c571f5d7-6f18-41f5-a33d-1e82e11bff52 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '38ffa549cacd4af29c9331077a5eca58'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:22.117690 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-c571f5d7-6f18-41f5-a33d-1e82e11bff52 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '38ffa549cacd4af29c9331077a5eca58'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:22.130855 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-c571f5d7-6f18-41f5-a33d-1e82e11bff52 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '38ffa549cacd4af29c9331077a5eca58'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:22.138732 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-c571f5d7-6f18-41f5-a33d-1e82e11bff52 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '38ffa549cacd4af29c9331077a5eca58'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:22.139482 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-c571f5d7-6f18-41f5-a33d-1e82e11bff52 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '38ffa549cacd4af29c9331077a5eca58'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:22.146323 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1070/2144] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:07:21 2022] DELETE /identity/v3/users/38ffa549cacd4af29c9331077a5eca58 => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:07:22.151175 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ffb4ef3-7b7d-4644-9dba-7e105125a31f None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:22.151463 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ffb4ef3-7b7d-4644-9dba-7e105125a31f None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:22.151731 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ffb4ef3-7b7d-4644-9dba-7e105125a31f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:22.175060 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-4ffb4ef3-7b7d-4644-9dba-7e105125a31f None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:07:22.176269 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-4ffb4ef3-7b7d-4644-9dba-7e105125a31f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:07:22.238758 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1075/2145] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:07:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:07:22.248512 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c571f5d7-6f18-41f5-a33d-1e82e11bff52 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:22.249939 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-c571f5d7-6f18-41f5-a33d-1e82e11bff52 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:07:22.309707 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c571f5d7-6f18-41f5-a33d-1e82e11bff52 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:22.313382 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5453be5-80c0-455b-86b8-07284dc348bb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:22.313981 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5453be5-80c0-455b-86b8-07284dc348bb service neutron] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:22.314258 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5453be5-80c0-455b-86b8-07284dc348bb service neutron] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:22.314556 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5453be5-80c0-455b-86b8-07284dc348bb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:22.323483 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5453be5-80c0-455b-86b8-07284dc348bb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:22.324992 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5453be5-80c0-455b-86b8-07284dc348bb service neutron] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:22.334363 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1071/2146] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:07:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:07:22.607280 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ffb4ef3-7b7d-4644-9dba-7e105125a31f None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:22.608983 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-4ffb4ef3-7b7d-4644-9dba-7e105125a31f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:07:22.621913 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ffb4ef3-7b7d-4644-9dba-7e105125a31f None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:22.626677 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-febe38d1-afc8-4661-90da-7294beb98e5a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:22.627385 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-febe38d1-afc8-4661-90da-7294beb98e5a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:22.627711 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-febe38d1-afc8-4661-90da-7294beb98e5a admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:22.628045 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-febe38d1-afc8-4661-90da-7294beb98e5a admin admin] PATH_INFO: `/v3/projects/03478f9b92284bd7ac1f16ddeffb87f1` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:22.629454 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-febe38d1-afc8-4661-90da-7294beb98e5a admin admin] RBAC: Authorizing `identity:delete_project(project_id=03478f9b92284bd7ac1f16ddeffb87f1)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:22.631465 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-febe38d1-afc8-4661-90da-7294beb98e5a admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:22.682923 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-febe38d1-afc8-4661-90da-7294beb98e5a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '03478f9b92284bd7ac1f16ddeffb87f1', 'request_id': 'req-febe38d1-afc8-4661-90da-7294beb98e5a'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:22.689306 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-febe38d1-afc8-4661-90da-7294beb98e5a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '03478f9b92284bd7ac1f16ddeffb87f1', 'request_id': 'req-febe38d1-afc8-4661-90da-7294beb98e5a'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:22.700218 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-febe38d1-afc8-4661-90da-7294beb98e5a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '03478f9b92284bd7ac1f16ddeffb87f1', 'request_id': 'req-febe38d1-afc8-4661-90da-7294beb98e5a'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:22.713710 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-febe38d1-afc8-4661-90da-7294beb98e5a admin admin] The token cache is being invalidate because project 03478f9b92284bd7ac1f16ddeffb87f1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81561) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:07:22.713997 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-febe38d1-afc8-4661-90da-7294beb98e5a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:22.714655 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-febe38d1-afc8-4661-90da-7294beb98e5a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:22.716769 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1076/2147] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:07:22 2022] DELETE /identity/v3/projects/03478f9b92284bd7ac1f16ddeffb87f1 => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:07:22.723250 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c353214-f2af-4231-8e6e-6aed7d795b4e None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:22.723743 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c353214-f2af-4231-8e6e-6aed7d795b4e None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:22.723929 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c353214-f2af-4231-8e6e-6aed7d795b4e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:22.743998 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-9c353214-f2af-4231-8e6e-6aed7d795b4e None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:07:22.745261 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-9c353214-f2af-4231-8e6e-6aed7d795b4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:07:22.803806 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1072/2148] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:07:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:07:22.812387 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-228ee9cd-498c-49fb-aa59-03c8127ec8bd None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:22.812877 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-228ee9cd-498c-49fb-aa59-03c8127ec8bd None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:22.812976 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-228ee9cd-498c-49fb-aa59-03c8127ec8bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:22.834109 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-228ee9cd-498c-49fb-aa59-03c8127ec8bd None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:07:22.835321 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-228ee9cd-498c-49fb-aa59-03c8127ec8bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:07:22.843656 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1077/2149] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:07:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:07:22.847747 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c353214-f2af-4231-8e6e-6aed7d795b4e None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:22.857684 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c353214-f2af-4231-8e6e-6aed7d795b4e None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:22.862221 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b28d5ba-a9b3-4bc8-89b5-d8ddb03125a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:22.862838 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b28d5ba-a9b3-4bc8-89b5-d8ddb03125a2 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:22.863115 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b28d5ba-a9b3-4bc8-89b5-d8ddb03125a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:22.863385 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b28d5ba-a9b3-4bc8-89b5-d8ddb03125a2 admin admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:22.864019 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b28d5ba-a9b3-4bc8-89b5-d8ddb03125a2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 09:07:22.864320 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b28d5ba-a9b3-4bc8-89b5-d8ddb03125a2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:22.865906 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b28d5ba-a9b3-4bc8-89b5-d8ddb03125a2 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:22.877866 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1073/2150] 10.222.0.36 () {58 vars in 1153 bytes} [Thu Oct 20 09:07:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 09:07:22.882022 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-228ee9cd-498c-49fb-aa59-03c8127ec8bd None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:22.891163 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-228ee9cd-498c-49fb-aa59-03c8127ec8bd None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:22.895767 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bcb28f1-7fc3-487a-9031-0e44414bbc8c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:22.896374 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bcb28f1-7fc3-487a-9031-0e44414bbc8c admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:22.896649 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bcb28f1-7fc3-487a-9031-0e44414bbc8c admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:22.896915 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bcb28f1-7fc3-487a-9031-0e44414bbc8c admin admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:22.897544 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bcb28f1-7fc3-487a-9031-0e44414bbc8c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:22.899461 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bcb28f1-7fc3-487a-9031-0e44414bbc8c admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:22.920364 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:07:22.928508 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1078/2151] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:07:22 2022] POST /identity/v3/projects => generated 373 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:07:22.932391 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b28d5ba-a9b3-4bc8-89b5-d8ddb03125a2 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:22.946954 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b28d5ba-a9b3-4bc8-89b5-d8ddb03125a2 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:22.955324 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:07:23.004196 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3566564e-a80a-4527-ae39-8d63cba1aaae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:23.004785 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3566564e-a80a-4527-ae39-8d63cba1aaae admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:23.005059 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3566564e-a80a-4527-ae39-8d63cba1aaae admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:23.005324 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3566564e-a80a-4527-ae39-8d63cba1aaae admin admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:23.005943 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3566564e-a80a-4527-ae39-8d63cba1aaae admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:23.007865 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3566564e-a80a-4527-ae39-8d63cba1aaae admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:23.028616 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1074/2152] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:07:22 2022] POST /identity/v3/users => generated 287 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:07:23.032724 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bcb28f1-7fc3-487a-9031-0e44414bbc8c admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:23.042988 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bcb28f1-7fc3-487a-9031-0e44414bbc8c admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:23.047585 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b9d67ba-d228-4b1a-a320-8039358920e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:23.048169 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b9d67ba-d228-4b1a-a320-8039358920e9 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:23.048445 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b9d67ba-d228-4b1a-a320-8039358920e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:23.048712 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b9d67ba-d228-4b1a-a320-8039358920e9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:23.049343 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b9d67ba-d228-4b1a-a320-8039358920e9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:23.051071 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b9d67ba-d228-4b1a-a320-8039358920e9 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:23.061878 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1079/2153] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:07:23 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:07:23.065613 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3566564e-a80a-4527-ae39-8d63cba1aaae admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:23.075840 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3566564e-a80a-4527-ae39-8d63cba1aaae admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:23.080476 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea8cb992-3c7e-4893-a045-cff434bace69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:23.081056 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea8cb992-3c7e-4893-a045-cff434bace69 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:23.081361 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea8cb992-3c7e-4893-a045-cff434bace69 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:23.081632 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea8cb992-3c7e-4893-a045-cff434bace69 admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:23.082299 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea8cb992-3c7e-4893-a045-cff434bace69 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:23.084960 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea8cb992-3c7e-4893-a045-cff434bace69 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:23.095767 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1075/2154] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:07:23 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:07:23.099736 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b9d67ba-d228-4b1a-a320-8039358920e9 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:23.109542 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b9d67ba-d228-4b1a-a320-8039358920e9 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:23.113994 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-546598dd-e706-457e-bd2d-96f7f66c76d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:23.114649 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-546598dd-e706-457e-bd2d-96f7f66c76d5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:23.114925 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-546598dd-e706-457e-bd2d-96f7f66c76d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:23.115194 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-546598dd-e706-457e-bd2d-96f7f66c76d5 admin admin] PATH_INFO: `/v3/projects/45d1505e085847329f7480c9bc08f1d8/users/21311bc5eef742f5a9eb81462a701ceb/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:23.129974 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-546598dd-e706-457e-bd2d-96f7f66c76d5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=45d1505e085847329f7480c9bc08f1d8, user_id=21311bc5eef742f5a9eb81462a701ceb, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:23.132660 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-546598dd-e706-457e-bd2d-96f7f66c76d5 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:23.145887 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1080/2155] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:07:23 2022] PUT /identity/v3/projects/45d1505e085847329f7480c9bc08f1d8/users/21311bc5eef742f5a9eb81462a701ceb/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:07:23.150886 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4923545-64d2-4d85-94eb-98f8ca0640e7 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:23.151194 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4923545-64d2-4d85-94eb-98f8ca0640e7 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:23.151460 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4923545-64d2-4d85-94eb-98f8ca0640e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:23.170275 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-a4923545-64d2-4d85-94eb-98f8ca0640e7 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:07:23.171555 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-a4923545-64d2-4d85-94eb-98f8ca0640e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:07:23.231806 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1076/2156] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:07:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 20 09:07:23.241310 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-546598dd-e706-457e-bd2d-96f7f66c76d5 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:23.243014 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-546598dd-e706-457e-bd2d-96f7f66c76d5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:07:23.300278 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-546598dd-e706-457e-bd2d-96f7f66c76d5 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:23.303874 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beb02d86-ba6b-4ec5-b3c0-33679e0bc40d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:23.304464 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb02d86-ba6b-4ec5-b3c0-33679e0bc40d service neutron] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:23.304736 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb02d86-ba6b-4ec5-b3c0-33679e0bc40d service neutron] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:23.305001 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb02d86-ba6b-4ec5-b3c0-33679e0bc40d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:23.313885 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beb02d86-ba6b-4ec5-b3c0-33679e0bc40d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:23.315338 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beb02d86-ba6b-4ec5-b3c0-33679e0bc40d service neutron] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:23.324576 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1081/2157] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:07:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:07:27.659064 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae85cde-653e-496a-8bfc-62e84fe38ecb None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:27.659555 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae85cde-653e-496a-8bfc-62e84fe38ecb None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:27.660148 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae85cde-653e-496a-8bfc-62e84fe38ecb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:27.693031 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-bae85cde-653e-496a-8bfc-62e84fe38ecb None None] MFA Rules not processed for user `21311bc5eef742f5a9eb81462a701ceb`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:07:27.694591 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-bae85cde-653e-496a-8bfc-62e84fe38ecb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:07:27.776621 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1077/2158] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:07:27 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:07:27.781840 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beb02d86-ba6b-4ec5-b3c0-33679e0bc40d service neutron] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:27.792079 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beb02d86-ba6b-4ec5-b3c0-33679e0bc40d service neutron] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:27.797551 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76ac63bd-8bcf-41fc-a634-c854325b3aed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:27.798176 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ac63bd-8bcf-41fc-a634-c854325b3aed admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:27.798454 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ac63bd-8bcf-41fc-a634-c854325b3aed admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:27.798758 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ac63bd-8bcf-41fc-a634-c854325b3aed admin admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:27.799412 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76ac63bd-8bcf-41fc-a634-c854325b3aed admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:27.801357 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76ac63bd-8bcf-41fc-a634-c854325b3aed admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:27.831827 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1082/2159] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:07:27 2022] POST /identity/v3/projects => generated 371 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:07:27.836177 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bae85cde-653e-496a-8bfc-62e84fe38ecb None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:27.848230 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bae85cde-653e-496a-8bfc-62e84fe38ecb None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:27.866166 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:07:27.911195 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f5d11c1-cd59-4d88-a27b-f4c91afee50a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:27.912450 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5d11c1-cd59-4d88-a27b-f4c91afee50a admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:27.912730 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5d11c1-cd59-4d88-a27b-f4c91afee50a admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:27.913001 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5d11c1-cd59-4d88-a27b-f4c91afee50a admin admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:27.913640 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f5d11c1-cd59-4d88-a27b-f4c91afee50a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:27.915643 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f5d11c1-cd59-4d88-a27b-f4c91afee50a admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:27.938911 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1078/2160] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:07:27 2022] POST /identity/v3/users => generated 292 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:07:27.942895 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76ac63bd-8bcf-41fc-a634-c854325b3aed admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:27.953376 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76ac63bd-8bcf-41fc-a634-c854325b3aed admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:27.958057 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b80d146-4371-4ee9-a258-8b200100c109 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:27.958694 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b80d146-4371-4ee9-a258-8b200100c109 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:27.959002 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b80d146-4371-4ee9-a258-8b200100c109 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:27.959271 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b80d146-4371-4ee9-a258-8b200100c109 admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:27.959907 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b80d146-4371-4ee9-a258-8b200100c109 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:27.961657 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b80d146-4371-4ee9-a258-8b200100c109 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:27.972882 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1083/2161] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:07:27 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:07:27.977045 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f5d11c1-cd59-4d88-a27b-f4c91afee50a admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:27.987527 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f5d11c1-cd59-4d88-a27b-f4c91afee50a admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:27.992154 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba9469b9-4cff-487c-8360-bbddb5554152 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:27.992809 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9469b9-4cff-487c-8360-bbddb5554152 admin admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:27.993089 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9469b9-4cff-487c-8360-bbddb5554152 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:27.993371 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9469b9-4cff-487c-8360-bbddb5554152 admin admin] PATH_INFO: `/v3/projects/e216ca9a35ba499998cd5a7a1a8b37c7/users/b75a8da9cea842ec8a1eff8314e5d46f/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:28.009728 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba9469b9-4cff-487c-8360-bbddb5554152 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e216ca9a35ba499998cd5a7a1a8b37c7, user_id=b75a8da9cea842ec8a1eff8314e5d46f, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:28.013166 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba9469b9-4cff-487c-8360-bbddb5554152 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:28.033552 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1079/2162] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:07:27 2022] PUT /identity/v3/projects/e216ca9a35ba499998cd5a7a1a8b37c7/users/b75a8da9cea842ec8a1eff8314e5d46f/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:07:28.035600 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b80d146-4371-4ee9-a258-8b200100c109 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:28.048181 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b80d146-4371-4ee9-a258-8b200100c109 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:28.127460 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcef86f6-776e-42f5-b43c-07d6fa4899bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:28.128157 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcef86f6-776e-42f5-b43c-07d6fa4899bd admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:28.128509 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcef86f6-776e-42f5-b43c-07d6fa4899bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:28.128926 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcef86f6-776e-42f5-b43c-07d6fa4899bd admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:28.129629 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcef86f6-776e-42f5-b43c-07d6fa4899bd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:28.131402 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcef86f6-776e-42f5-b43c-07d6fa4899bd admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:28.152215 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1084/2163] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:07:28 2022] GET /identity/v3/roles => generated 1420 bytes in 117 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:07:28.156173 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba9469b9-4cff-487c-8360-bbddb5554152 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:28.169384 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba9469b9-4cff-487c-8360-bbddb5554152 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:28.174627 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d292264a-1a89-4f72-a169-29ca09c05dee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:28.175282 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d292264a-1a89-4f72-a169-29ca09c05dee admin admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:28.175555 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d292264a-1a89-4f72-a169-29ca09c05dee admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:28.175816 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d292264a-1a89-4f72-a169-29ca09c05dee admin admin] PATH_INFO: `/v3/projects/e216ca9a35ba499998cd5a7a1a8b37c7/users/b75a8da9cea842ec8a1eff8314e5d46f/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:28.177689 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d292264a-1a89-4f72-a169-29ca09c05dee admin admin] RBAC: Authorizing `identity:create_grant(project_id=e216ca9a35ba499998cd5a7a1a8b37c7, user_id=b75a8da9cea842ec8a1eff8314e5d46f, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:28.180862 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d292264a-1a89-4f72-a169-29ca09c05dee admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:28.194020 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1080/2164] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:07:28 2022] PUT /identity/v3/projects/e216ca9a35ba499998cd5a7a1a8b37c7/users/b75a8da9cea842ec8a1eff8314e5d46f/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:07:35.722331 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831ef486-6fca-43fb-80be-46f42a66ec5e None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:35.723435 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831ef486-6fca-43fb-80be-46f42a66ec5e None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:35.723884 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831ef486-6fca-43fb-80be-46f42a66ec5e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:35.814245 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-831ef486-6fca-43fb-80be-46f42a66ec5e None None] MFA Rules not processed for user `b75a8da9cea842ec8a1eff8314e5d46f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:07:35.817325 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-831ef486-6fca-43fb-80be-46f42a66ec5e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:07:35.876802 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:07:36.041489 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1085/2165] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:07:35 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 320 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:07:36.055971 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d292264a-1a89-4f72-a169-29ca09c05dee admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:36.058621 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-d292264a-1a89-4f72-a169-29ca09c05dee admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:07:36.249548 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d292264a-1a89-4f72-a169-29ca09c05dee admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:36.267589 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0fca833-ee0c-4041-aa05-fca798654f60 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a94e868115ec49918eda10724263930c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:36.268552 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0fca833-ee0c-4041-aa05-fca798654f60 service nova] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:36.269092 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0fca833-ee0c-4041-aa05-fca798654f60 service nova] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:36.269477 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0fca833-ee0c-4041-aa05-fca798654f60 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:36.397274 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0fca833-ee0c-4041-aa05-fca798654f60 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:36.400101 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0fca833-ee0c-4041-aa05-fca798654f60 service nova] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:36.419057 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1081/2166] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:07:36 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 364 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:07:37.601693 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d389cb-04a7-43d1-8769-38e53e919690 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:37.602244 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d389cb-04a7-43d1-8769-38e53e919690 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:37.602365 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d389cb-04a7-43d1-8769-38e53e919690 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:37.622041 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-e4d389cb-04a7-43d1-8769-38e53e919690 None None] MFA Rules not processed for user `21311bc5eef742f5a9eb81462a701ceb`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:07:37.623300 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-e4d389cb-04a7-43d1-8769-38e53e919690 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:07:37.630192 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1086/2167] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:07:37 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:07:40.773034 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0fca833-ee0c-4041-aa05-fca798654f60 service nova] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:40.790352 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:07:40.837420 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0fca833-ee0c-4041-aa05-fca798654f60 service nova] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:40.842013 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9962aa26-727b-4524-a198-ec36cfc9622e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:40.842775 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9962aa26-727b-4524-a198-ec36cfc9622e service neutron] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:40.843079 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9962aa26-727b-4524-a198-ec36cfc9622e service neutron] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:40.843467 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9962aa26-727b-4524-a198-ec36cfc9622e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:40.845557 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-9962aa26-727b-4524-a198-ec36cfc9622e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:07:40.921455 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9962aa26-727b-4524-a198-ec36cfc9622e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:40.922957 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9962aa26-727b-4524-a198-ec36cfc9622e service neutron] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:40.933056 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1082/2168] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:07:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 161 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:07:44.505937 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d389cb-04a7-43d1-8769-38e53e919690 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:44.508942 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-e4d389cb-04a7-43d1-8769-38e53e919690 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:07:44.585486 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9962aa26-727b-4524-a198-ec36cfc9622e service neutron] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:44.587637 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-9962aa26-727b-4524-a198-ec36cfc9622e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:07:44.634981 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9962aa26-727b-4524-a198-ec36cfc9622e service neutron] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:44.651280 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06c29fc9-4ef4-40b7-be4c-c54493755b7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:44.652565 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c29fc9-4ef4-40b7-be4c-c54493755b7e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:44.654695 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c29fc9-4ef4-40b7-be4c-c54493755b7e admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:44.657586 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c29fc9-4ef4-40b7-be4c-c54493755b7e admin admin] PATH_INFO: `/v3/users/8002ab26f6d64822aaa65aa730781b76` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:44.663549 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06c29fc9-4ef4-40b7-be4c-c54493755b7e admin admin] RBAC: Authorizing `identity:delete_user(user_id=8002ab26f6d64822aaa65aa730781b76)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:44.677288 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06c29fc9-4ef4-40b7-be4c-c54493755b7e admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:44.696144 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d389cb-04a7-43d1-8769-38e53e919690 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:44.710740 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdebad2a-ed84-406e-b220-ae2f5d83023c service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ad03ab76b9c44b1cb369475afa25eddd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:44.711449 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdebad2a-ed84-406e-b220-ae2f5d83023c service placement] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:44.712374 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdebad2a-ed84-406e-b220-ae2f5d83023c service placement] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:44.712955 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdebad2a-ed84-406e-b220-ae2f5d83023c service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:44.718542 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-cdebad2a-ed84-406e-b220-ae2f5d83023c service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:07:44.751834 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdebad2a-ed84-406e-b220-ae2f5d83023c service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:44.761022 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdebad2a-ed84-406e-b220-ae2f5d83023c service placement] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:44.784284 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1087/2169] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:07:44 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 279 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:07:44.845876 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-06c29fc9-4ef4-40b7-be4c-c54493755b7e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8002ab26f6d64822aaa65aa730781b76'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:44.863829 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-06c29fc9-4ef4-40b7-be4c-c54493755b7e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8002ab26f6d64822aaa65aa730781b76'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:44.880887 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-06c29fc9-4ef4-40b7-be4c-c54493755b7e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8002ab26f6d64822aaa65aa730781b76'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:44.890815 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-06c29fc9-4ef4-40b7-be4c-c54493755b7e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8002ab26f6d64822aaa65aa730781b76'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:44.891594 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-06c29fc9-4ef4-40b7-be4c-c54493755b7e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8002ab26f6d64822aaa65aa730781b76'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:44.898786 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1083/2170] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:07:44 2022] DELETE /identity/v3/users/8002ab26f6d64822aaa65aa730781b76 => generated 0 bytes in 314 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:07:45.089683 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdebad2a-ed84-406e-b220-ae2f5d83023c service placement] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:45.102263 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdebad2a-ed84-406e-b220-ae2f5d83023c service placement] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:45.163929 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b187f763-0d4b-4dbf-89ca-ab0dde6f7e19 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:45.164660 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b187f763-0d4b-4dbf-89ca-ab0dde6f7e19 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:45.165004 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b187f763-0d4b-4dbf-89ca-ab0dde6f7e19 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:45.165341 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b187f763-0d4b-4dbf-89ca-ab0dde6f7e19 admin admin] PATH_INFO: `/v3/projects/fcf51428b6bd4c428e1e01aadc6abe48` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:45.166555 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b187f763-0d4b-4dbf-89ca-ab0dde6f7e19 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fcf51428b6bd4c428e1e01aadc6abe48)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:45.168553 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b187f763-0d4b-4dbf-89ca-ab0dde6f7e19 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:45.224038 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:07:45.232614 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-b187f763-0d4b-4dbf-89ca-ab0dde6f7e19 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fcf51428b6bd4c428e1e01aadc6abe48', 'request_id': 'req-b187f763-0d4b-4dbf-89ca-ab0dde6f7e19'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:45.246581 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-b187f763-0d4b-4dbf-89ca-ab0dde6f7e19 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fcf51428b6bd4c428e1e01aadc6abe48', 'request_id': 'req-b187f763-0d4b-4dbf-89ca-ab0dde6f7e19'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:45.253911 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-b187f763-0d4b-4dbf-89ca-ab0dde6f7e19 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fcf51428b6bd4c428e1e01aadc6abe48', 'request_id': 'req-b187f763-0d4b-4dbf-89ca-ab0dde6f7e19'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:45.284704 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-b187f763-0d4b-4dbf-89ca-ab0dde6f7e19 admin admin] The token cache is being invalidate because project fcf51428b6bd4c428e1e01aadc6abe48 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81561) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:07:45.284704 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-b187f763-0d4b-4dbf-89ca-ab0dde6f7e19 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:45.285404 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-b187f763-0d4b-4dbf-89ca-ab0dde6f7e19 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:45.287513 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1088/2171] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:07:45 2022] DELETE /identity/v3/projects/fcf51428b6bd4c428e1e01aadc6abe48 => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:07:45.292663 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81485acc-d3ea-4158-87bb-7fa9c2fe4da2 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:45.293028 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81485acc-d3ea-4158-87bb-7fa9c2fe4da2 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:45.293371 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81485acc-d3ea-4158-87bb-7fa9c2fe4da2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:45.316257 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-81485acc-d3ea-4158-87bb-7fa9c2fe4da2 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:07:45.317478 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-81485acc-d3ea-4158-87bb-7fa9c2fe4da2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:07:45.345649 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:07:45.390702 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1084/2172] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:07:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:07:45.398430 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddbf26e6-33e5-4a8c-a46d-759b3610ec72 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:45.398751 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddbf26e6-33e5-4a8c-a46d-759b3610ec72 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:45.399027 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddbf26e6-33e5-4a8c-a46d-759b3610ec72 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:45.423386 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-ddbf26e6-33e5-4a8c-a46d-759b3610ec72 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:07:45.423386 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-ddbf26e6-33e5-4a8c-a46d-759b3610ec72 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:07:45.430122 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1089/2173] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:07:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:07:45.434312 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81485acc-d3ea-4158-87bb-7fa9c2fe4da2 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:45.445036 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81485acc-d3ea-4158-87bb-7fa9c2fe4da2 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:45.449680 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f886b05c-5eaf-4b72-ac4c-4b7b82adf854 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:45.450282 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f886b05c-5eaf-4b72-ac4c-4b7b82adf854 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:45.451143 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f886b05c-5eaf-4b72-ac4c-4b7b82adf854 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:45.451418 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f886b05c-5eaf-4b72-ac4c-4b7b82adf854 admin admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:45.452064 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f886b05c-5eaf-4b72-ac4c-4b7b82adf854 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 09:07:45.452360 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f886b05c-5eaf-4b72-ac4c-4b7b82adf854 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:45.453970 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f886b05c-5eaf-4b72-ac4c-4b7b82adf854 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:45.467411 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1085/2174] 10.222.0.36 () {58 vars in 1153 bytes} [Thu Oct 20 09:07:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 09:07:45.471904 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddbf26e6-33e5-4a8c-a46d-759b3610ec72 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:45.481336 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddbf26e6-33e5-4a8c-a46d-759b3610ec72 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:45.486682 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79042bac-5c2f-46de-9f0d-e8bc2c7facd9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:45.487275 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79042bac-5c2f-46de-9f0d-e8bc2c7facd9 admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:45.487551 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79042bac-5c2f-46de-9f0d-e8bc2c7facd9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:45.487821 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79042bac-5c2f-46de-9f0d-e8bc2c7facd9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:45.488458 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79042bac-5c2f-46de-9f0d-e8bc2c7facd9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:45.490371 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79042bac-5c2f-46de-9f0d-e8bc2c7facd9 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:45.519223 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1090/2175] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:07:45 2022] POST /identity/v3/projects => generated 369 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:07:45.523549 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f886b05c-5eaf-4b72-ac4c-4b7b82adf854 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:45.534307 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f886b05c-5eaf-4b72-ac4c-4b7b82adf854 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:45.593050 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f349b93-9781-4dae-94b9-6f5e9e16e0a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:45.593758 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f349b93-9781-4dae-94b9-6f5e9e16e0a0 admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:45.593986 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f349b93-9781-4dae-94b9-6f5e9e16e0a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:45.594169 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f349b93-9781-4dae-94b9-6f5e9e16e0a0 admin admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:45.594878 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f349b93-9781-4dae-94b9-6f5e9e16e0a0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:45.596797 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f349b93-9781-4dae-94b9-6f5e9e16e0a0 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:45.620573 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1086/2176] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:07:45 2022] POST /identity/v3/users => generated 285 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 20 09:07:45.625112 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79042bac-5c2f-46de-9f0d-e8bc2c7facd9 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:45.639525 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79042bac-5c2f-46de-9f0d-e8bc2c7facd9 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:45.644264 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e17a5aac-5197-433b-94ac-8bb15f3307f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:45.644872 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17a5aac-5197-433b-94ac-8bb15f3307f9 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:45.645146 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17a5aac-5197-433b-94ac-8bb15f3307f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:45.645417 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17a5aac-5197-433b-94ac-8bb15f3307f9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:45.646052 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e17a5aac-5197-433b-94ac-8bb15f3307f9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:45.647691 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e17a5aac-5197-433b-94ac-8bb15f3307f9 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:45.659436 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1091/2177] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:07:45 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:07:45.663759 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f349b93-9781-4dae-94b9-6f5e9e16e0a0 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:45.673920 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f349b93-9781-4dae-94b9-6f5e9e16e0a0 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:45.678768 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60fc8b63-615a-489c-975d-c1e917a8d9af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:45.679366 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60fc8b63-615a-489c-975d-c1e917a8d9af admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:45.679652 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60fc8b63-615a-489c-975d-c1e917a8d9af admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:45.679919 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60fc8b63-615a-489c-975d-c1e917a8d9af admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:45.680660 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60fc8b63-615a-489c-975d-c1e917a8d9af admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:45.682266 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60fc8b63-615a-489c-975d-c1e917a8d9af admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:45.696520 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1087/2178] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:07:45 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:07:45.701314 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e17a5aac-5197-433b-94ac-8bb15f3307f9 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:45.711937 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e17a5aac-5197-433b-94ac-8bb15f3307f9 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:45.717455 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec8f291-6446-49c9-8780-43186e72185f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:45.718095 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec8f291-6446-49c9-8780-43186e72185f admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:45.718367 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec8f291-6446-49c9-8780-43186e72185f admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:45.718664 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec8f291-6446-49c9-8780-43186e72185f admin admin] PATH_INFO: `/v3/projects/4a8df23cfe9b4c779974847098bb6a8c/users/1c11f96690f04580be2c330b3f57dcc3/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:45.734311 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cec8f291-6446-49c9-8780-43186e72185f admin admin] RBAC: Authorizing `identity:create_grant(project_id=4a8df23cfe9b4c779974847098bb6a8c, user_id=1c11f96690f04580be2c330b3f57dcc3, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:45.737107 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cec8f291-6446-49c9-8780-43186e72185f admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:45.752068 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1092/2179] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:07:45 2022] PUT /identity/v3/projects/4a8df23cfe9b4c779974847098bb6a8c/users/1c11f96690f04580be2c330b3f57dcc3/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:07:45.763932 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14e4491-7906-425c-bd50-81d719f23665 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:45.764218 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14e4491-7906-425c-bd50-81d719f23665 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:45.764492 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14e4491-7906-425c-bd50-81d719f23665 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:45.798920 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-f14e4491-7906-425c-bd50-81d719f23665 None None] MFA Rules not processed for user `1c11f96690f04580be2c330b3f57dcc3`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:07:45.800229 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-f14e4491-7906-425c-bd50-81d719f23665 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:07:45.873580 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1088/2180] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:07:45 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:07:45.885121 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec8f291-6446-49c9-8780-43186e72185f admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:45.886962 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-cec8f291-6446-49c9-8780-43186e72185f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:07:45.965680 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec8f291-6446-49c9-8780-43186e72185f admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:45.970809 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76ba9c95-a6fb-4aba-9d8f-af3ec288f242 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a94e868115ec49918eda10724263930c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:45.971478 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ba9c95-a6fb-4aba-9d8f-af3ec288f242 service nova] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:45.971818 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ba9c95-a6fb-4aba-9d8f-af3ec288f242 service nova] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:45.972185 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ba9c95-a6fb-4aba-9d8f-af3ec288f242 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:45.980323 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76ba9c95-a6fb-4aba-9d8f-af3ec288f242 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:45.982085 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76ba9c95-a6fb-4aba-9d8f-af3ec288f242 service nova] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:45.991549 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1093/2181] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:07:45 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:07:46.208421 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f14e4491-7906-425c-bd50-81d719f23665 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:46.238951 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:07:46.277801 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f14e4491-7906-425c-bd50-81d719f23665 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:46.282847 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-469832cf-d031-4529-b53b-562cf39eec57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:46.283476 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-469832cf-d031-4529-b53b-562cf39eec57 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:46.283811 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-469832cf-d031-4529-b53b-562cf39eec57 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:46.284086 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-469832cf-d031-4529-b53b-562cf39eec57 admin admin] PATH_INFO: `/v3/users/1c11f96690f04580be2c330b3f57dcc3` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:46.285277 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-469832cf-d031-4529-b53b-562cf39eec57 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1c11f96690f04580be2c330b3f57dcc3)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:46.287296 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-469832cf-d031-4529-b53b-562cf39eec57 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:46.335378 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-469832cf-d031-4529-b53b-562cf39eec57 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1c11f96690f04580be2c330b3f57dcc3'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:46.343709 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-469832cf-d031-4529-b53b-562cf39eec57 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1c11f96690f04580be2c330b3f57dcc3'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:46.355928 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-469832cf-d031-4529-b53b-562cf39eec57 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1c11f96690f04580be2c330b3f57dcc3'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:46.363414 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-469832cf-d031-4529-b53b-562cf39eec57 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1c11f96690f04580be2c330b3f57dcc3'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:46.364070 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-469832cf-d031-4529-b53b-562cf39eec57 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1c11f96690f04580be2c330b3f57dcc3'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:46.370506 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1089/2182] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:07:46 2022] DELETE /identity/v3/users/1c11f96690f04580be2c330b3f57dcc3 => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:07:46.375560 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a4213f-a82d-4e5a-8feb-623abad1b527 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:46.375853 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a4213f-a82d-4e5a-8feb-623abad1b527 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:46.376126 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a4213f-a82d-4e5a-8feb-623abad1b527 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:46.398273 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-87a4213f-a82d-4e5a-8feb-623abad1b527 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:07:46.399522 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-87a4213f-a82d-4e5a-8feb-623abad1b527 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:07:46.445857 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=93,keystone:DELETE=4,keystone:INSERT=3 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:07:46.464421 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1094/2183] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:07:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:07:46.474399 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-469832cf-d031-4529-b53b-562cf39eec57 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:46.475978 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-469832cf-d031-4529-b53b-562cf39eec57 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:07:46.544170 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-469832cf-d031-4529-b53b-562cf39eec57 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:46.548879 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66a6301f-0506-4d91-9a9b-74e6e7a78d0d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:46.549501 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66a6301f-0506-4d91-9a9b-74e6e7a78d0d service neutron] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:46.549782 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66a6301f-0506-4d91-9a9b-74e6e7a78d0d service neutron] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:46.550058 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66a6301f-0506-4d91-9a9b-74e6e7a78d0d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:46.559805 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66a6301f-0506-4d91-9a9b-74e6e7a78d0d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:46.561292 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66a6301f-0506-4d91-9a9b-74e6e7a78d0d service neutron] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:46.572009 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1090/2184] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:07:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 20 09:07:46.874206 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87a4213f-a82d-4e5a-8feb-623abad1b527 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:46.875708 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-87a4213f-a82d-4e5a-8feb-623abad1b527 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:07:46.887987 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87a4213f-a82d-4e5a-8feb-623abad1b527 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:46.893462 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7571829c-056c-4590-87e5-1f283af92f3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:46.894086 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7571829c-056c-4590-87e5-1f283af92f3c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:46.894368 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7571829c-056c-4590-87e5-1f283af92f3c admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:46.894820 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7571829c-056c-4590-87e5-1f283af92f3c admin admin] PATH_INFO: `/v3/projects/4a8df23cfe9b4c779974847098bb6a8c` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:46.895940 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7571829c-056c-4590-87e5-1f283af92f3c admin admin] RBAC: Authorizing `identity:delete_project(project_id=4a8df23cfe9b4c779974847098bb6a8c)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:46.897855 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7571829c-056c-4590-87e5-1f283af92f3c admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:46.947293 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-7571829c-056c-4590-87e5-1f283af92f3c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4a8df23cfe9b4c779974847098bb6a8c', 'request_id': 'req-7571829c-056c-4590-87e5-1f283af92f3c'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:46.953615 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-7571829c-056c-4590-87e5-1f283af92f3c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4a8df23cfe9b4c779974847098bb6a8c', 'request_id': 'req-7571829c-056c-4590-87e5-1f283af92f3c'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:46.959424 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-7571829c-056c-4590-87e5-1f283af92f3c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4a8df23cfe9b4c779974847098bb6a8c', 'request_id': 'req-7571829c-056c-4590-87e5-1f283af92f3c'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:46.971935 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-7571829c-056c-4590-87e5-1f283af92f3c admin admin] The token cache is being invalidate because project 4a8df23cfe9b4c779974847098bb6a8c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81561) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:07:46.972222 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-7571829c-056c-4590-87e5-1f283af92f3c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:46.972853 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-7571829c-056c-4590-87e5-1f283af92f3c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:46.974672 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1095/2185] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:07:46 2022] DELETE /identity/v3/projects/4a8df23cfe9b4c779974847098bb6a8c => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:07:46.979581 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23216cf9-d22f-4e18-adbc-785bbbdfec3b None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:46.980097 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23216cf9-d22f-4e18-adbc-785bbbdfec3b None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:46.980097 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23216cf9-d22f-4e18-adbc-785bbbdfec3b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:47.000881 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-23216cf9-d22f-4e18-adbc-785bbbdfec3b None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:07:47.002111 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-23216cf9-d22f-4e18-adbc-785bbbdfec3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:07:47.051363 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:07:47.070245 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1091/2186] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:07:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:07:47.078462 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37473d42-e391-4963-a3b8-9e19f7b20bb8 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:47.078462 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37473d42-e391-4963-a3b8-9e19f7b20bb8 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:47.078462 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37473d42-e391-4963-a3b8-9e19f7b20bb8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:47.101505 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-37473d42-e391-4963-a3b8-9e19f7b20bb8 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:07:47.102775 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-37473d42-e391-4963-a3b8-9e19f7b20bb8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:07:47.111322 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1096/2187] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:07:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:07:47.115460 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23216cf9-d22f-4e18-adbc-785bbbdfec3b None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:47.125086 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23216cf9-d22f-4e18-adbc-785bbbdfec3b None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:47.132516 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-470a400a-cd39-4cfb-b1a0-7b3012af0698 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:47.133142 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470a400a-cd39-4cfb-b1a0-7b3012af0698 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:47.133427 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470a400a-cd39-4cfb-b1a0-7b3012af0698 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:47.133697 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470a400a-cd39-4cfb-b1a0-7b3012af0698 admin admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:47.134353 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-470a400a-cd39-4cfb-b1a0-7b3012af0698 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 09:07:47.135299 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-470a400a-cd39-4cfb-b1a0-7b3012af0698 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:47.137300 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-470a400a-cd39-4cfb-b1a0-7b3012af0698 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:47.155246 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1092/2188] 10.222.0.36 () {58 vars in 1153 bytes} [Thu Oct 20 09:07:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 09:07:47.160449 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37473d42-e391-4963-a3b8-9e19f7b20bb8 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:47.170910 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37473d42-e391-4963-a3b8-9e19f7b20bb8 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:47.175699 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d78267cb-8168-4c6a-8eee-215ac9347534 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:47.176312 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d78267cb-8168-4c6a-8eee-215ac9347534 admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:47.176589 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d78267cb-8168-4c6a-8eee-215ac9347534 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:47.176856 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d78267cb-8168-4c6a-8eee-215ac9347534 admin admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:47.177493 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d78267cb-8168-4c6a-8eee-215ac9347534 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:47.179665 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d78267cb-8168-4c6a-8eee-215ac9347534 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:47.209120 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1097/2189] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:07:47 2022] POST /identity/v3/projects => generated 377 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:07:47.213230 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-470a400a-cd39-4cfb-b1a0-7b3012af0698 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:47.224032 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-470a400a-cd39-4cfb-b1a0-7b3012af0698 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:47.282011 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b51779ef-867b-435f-940f-4274fb2d05d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:47.282813 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b51779ef-867b-435f-940f-4274fb2d05d4 admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:47.283092 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b51779ef-867b-435f-940f-4274fb2d05d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:47.283360 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b51779ef-867b-435f-940f-4274fb2d05d4 admin admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:47.283982 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b51779ef-867b-435f-940f-4274fb2d05d4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:47.285920 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b51779ef-867b-435f-940f-4274fb2d05d4 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:47.314101 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1093/2190] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:07:47 2022] POST /identity/v3/users => generated 289 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:07:47.318142 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d78267cb-8168-4c6a-8eee-215ac9347534 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:47.328785 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d78267cb-8168-4c6a-8eee-215ac9347534 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:47.333599 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93289670-734b-4a7a-8ff0-78f4f8f1e659 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:47.334221 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93289670-734b-4a7a-8ff0-78f4f8f1e659 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:47.334547 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93289670-734b-4a7a-8ff0-78f4f8f1e659 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:47.334824 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93289670-734b-4a7a-8ff0-78f4f8f1e659 admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:47.335637 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93289670-734b-4a7a-8ff0-78f4f8f1e659 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:47.337324 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93289670-734b-4a7a-8ff0-78f4f8f1e659 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:47.349221 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1098/2191] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:07:47 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 20 09:07:47.353082 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b51779ef-867b-435f-940f-4274fb2d05d4 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:47.366719 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b51779ef-867b-435f-940f-4274fb2d05d4 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:47.371898 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe75a5e2-26e0-42a5-85f8-a98c92069800 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:47.372580 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe75a5e2-26e0-42a5-85f8-a98c92069800 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:47.372880 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe75a5e2-26e0-42a5-85f8-a98c92069800 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:47.373149 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe75a5e2-26e0-42a5-85f8-a98c92069800 admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:47.373878 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe75a5e2-26e0-42a5-85f8-a98c92069800 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:47.375662 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe75a5e2-26e0-42a5-85f8-a98c92069800 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:47.393949 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1094/2192] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:07:47 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:07:47.395150 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93289670-734b-4a7a-8ff0-78f4f8f1e659 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:47.406569 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93289670-734b-4a7a-8ff0-78f4f8f1e659 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:47.411266 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe7e938c-d746-4c77-8fbf-51c47f7f1a89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:47.412143 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe7e938c-d746-4c77-8fbf-51c47f7f1a89 admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:47.412271 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe7e938c-d746-4c77-8fbf-51c47f7f1a89 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:47.412510 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe7e938c-d746-4c77-8fbf-51c47f7f1a89 admin admin] PATH_INFO: `/v3/projects/52bb2568c8f04a1ba1d87a0d89c4bd05/users/b52bc47bcce84c0c81aad4172106c04a/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:47.428251 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe7e938c-d746-4c77-8fbf-51c47f7f1a89 admin admin] RBAC: Authorizing `identity:create_grant(project_id=52bb2568c8f04a1ba1d87a0d89c4bd05, user_id=b52bc47bcce84c0c81aad4172106c04a, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:47.431019 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe7e938c-d746-4c77-8fbf-51c47f7f1a89 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:47.444729 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1099/2193] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:07:47 2022] PUT /identity/v3/projects/52bb2568c8f04a1ba1d87a0d89c4bd05/users/b52bc47bcce84c0c81aad4172106c04a/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:07:47.456037 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db02d3d-9463-4fc5-9c20-aca0e3934b51 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:47.456397 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db02d3d-9463-4fc5-9c20-aca0e3934b51 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:47.456666 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db02d3d-9463-4fc5-9c20-aca0e3934b51 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:47.487834 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-8db02d3d-9463-4fc5-9c20-aca0e3934b51 None None] MFA Rules not processed for user `b52bc47bcce84c0c81aad4172106c04a`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:07:47.488961 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-8db02d3d-9463-4fc5-9c20-aca0e3934b51 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:07:47.562407 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1095/2194] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:07:47 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:07:47.566826 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe7e938c-d746-4c77-8fbf-51c47f7f1a89 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:47.633184 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe7e938c-d746-4c77-8fbf-51c47f7f1a89 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:47.641437 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3bfcda8-f446-4955-929b-1bde1f6ee547 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:47.642054 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3bfcda8-f446-4955-929b-1bde1f6ee547 admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:47.642331 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3bfcda8-f446-4955-929b-1bde1f6ee547 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:47.642639 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3bfcda8-f446-4955-929b-1bde1f6ee547 admin admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:47.643831 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3bfcda8-f446-4955-929b-1bde1f6ee547 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:47.645774 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3bfcda8-f446-4955-929b-1bde1f6ee547 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:47.670943 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:07:47.688291 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1100/2195] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:07:47 2022] POST /identity/v3/projects => generated 377 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:07:47.692411 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8db02d3d-9463-4fc5-9c20-aca0e3934b51 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:47.704245 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8db02d3d-9463-4fc5-9c20-aca0e3934b51 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:47.752635 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:07:47.775224 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4051156e-6a12-4387-b3b3-b13524ca0918 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:47.775839 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4051156e-6a12-4387-b3b3-b13524ca0918 admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:47.776117 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4051156e-6a12-4387-b3b3-b13524ca0918 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:47.776384 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4051156e-6a12-4387-b3b3-b13524ca0918 admin admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:47.777026 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4051156e-6a12-4387-b3b3-b13524ca0918 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:47.779060 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4051156e-6a12-4387-b3b3-b13524ca0918 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:47.809326 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1096/2196] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:07:47 2022] POST /identity/v3/users => generated 295 bytes in 117 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 20 09:07:47.813668 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3bfcda8-f446-4955-929b-1bde1f6ee547 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:47.824820 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3bfcda8-f446-4955-929b-1bde1f6ee547 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:47.830592 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8f6d711-a3b6-4de2-b4f8-9519626d4a3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:47.831224 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f6d711-a3b6-4de2-b4f8-9519626d4a3e admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:47.831573 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f6d711-a3b6-4de2-b4f8-9519626d4a3e admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:47.831947 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f6d711-a3b6-4de2-b4f8-9519626d4a3e admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:47.832699 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8f6d711-a3b6-4de2-b4f8-9519626d4a3e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:47.834517 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8f6d711-a3b6-4de2-b4f8-9519626d4a3e admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:47.850502 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1101/2197] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:07:47 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:07:47.851705 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4051156e-6a12-4387-b3b3-b13524ca0918 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:47.864983 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4051156e-6a12-4387-b3b3-b13524ca0918 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:47.870031 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-108e778f-0c53-4aeb-82d9-7c26caa999eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:47.870748 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-108e778f-0c53-4aeb-82d9-7c26caa999eb admin admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:47.871029 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-108e778f-0c53-4aeb-82d9-7c26caa999eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:47.871305 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-108e778f-0c53-4aeb-82d9-7c26caa999eb admin admin] PATH_INFO: `/v3/projects/61ee7d5324e84ec9990f5d2bd946f343/users/4869f672ff9e4a17a378b49a984f628c/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:47.890936 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-108e778f-0c53-4aeb-82d9-7c26caa999eb admin admin] RBAC: Authorizing `identity:create_grant(project_id=61ee7d5324e84ec9990f5d2bd946f343, user_id=4869f672ff9e4a17a378b49a984f628c, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:47.894032 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-108e778f-0c53-4aeb-82d9-7c26caa999eb admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:47.909941 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1097/2198] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:07:47 2022] PUT /identity/v3/projects/61ee7d5324e84ec9990f5d2bd946f343/users/4869f672ff9e4a17a378b49a984f628c/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:07:47.914117 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8f6d711-a3b6-4de2-b4f8-9519626d4a3e admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:47.926942 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8f6d711-a3b6-4de2-b4f8-9519626d4a3e admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:47.991254 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77be3e26-0207-408c-bbbb-3e1be2fdaccd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:47.991871 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77be3e26-0207-408c-bbbb-3e1be2fdaccd admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:47.992152 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77be3e26-0207-408c-bbbb-3e1be2fdaccd admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:47.992424 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77be3e26-0207-408c-bbbb-3e1be2fdaccd admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:47.993207 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77be3e26-0207-408c-bbbb-3e1be2fdaccd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:47.995180 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77be3e26-0207-408c-bbbb-3e1be2fdaccd admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:48.010409 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1102/2199] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:07:47 2022] GET /identity/v3/roles => generated 1420 bytes in 94 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:07:48.011727 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-108e778f-0c53-4aeb-82d9-7c26caa999eb admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:48.023289 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-108e778f-0c53-4aeb-82d9-7c26caa999eb admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:48.028000 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ee4ae8-dddc-4076-8a3f-14ba9d011347 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:48.029201 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ee4ae8-dddc-4076-8a3f-14ba9d011347 admin admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:48.029524 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ee4ae8-dddc-4076-8a3f-14ba9d011347 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:48.029818 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ee4ae8-dddc-4076-8a3f-14ba9d011347 admin admin] PATH_INFO: `/v3/projects/61ee7d5324e84ec9990f5d2bd946f343/users/4869f672ff9e4a17a378b49a984f628c/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:48.031698 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66ee4ae8-dddc-4076-8a3f-14ba9d011347 admin admin] RBAC: Authorizing `identity:create_grant(project_id=61ee7d5324e84ec9990f5d2bd946f343, user_id=4869f672ff9e4a17a378b49a984f628c, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:48.034374 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66ee4ae8-dddc-4076-8a3f-14ba9d011347 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:48.047215 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1098/2200] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:07:48 2022] PUT /identity/v3/projects/61ee7d5324e84ec9990f5d2bd946f343/users/4869f672ff9e4a17a378b49a984f628c/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:07:48.058279 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b34e6377-31e7-4762-a514-6af5859dd5b3 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:48.058687 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b34e6377-31e7-4762-a514-6af5859dd5b3 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:48.059037 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b34e6377-31e7-4762-a514-6af5859dd5b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:48.090668 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-b34e6377-31e7-4762-a514-6af5859dd5b3 None None] MFA Rules not processed for user `4869f672ff9e4a17a378b49a984f628c`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:07:48.092405 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-b34e6377-31e7-4762-a514-6af5859dd5b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:07:48.191169 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1103/2201] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:07:48 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:07:48.199967 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ee4ae8-dddc-4076-8a3f-14ba9d011347 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:48.201899 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-66ee4ae8-dddc-4076-8a3f-14ba9d011347 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:07:48.228611 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20336808-8772-4a71-9ff1-bfee9e4c5146 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:48.228990 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20336808-8772-4a71-9ff1-bfee9e4c5146 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:48.229340 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20336808-8772-4a71-9ff1-bfee9e4c5146 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:48.261116 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-20336808-8772-4a71-9ff1-bfee9e4c5146 None None] MFA Rules not processed for user `f4b0e9187ce74f7888e9e7094be5c824`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:07:48.262154 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-20336808-8772-4a71-9ff1-bfee9e4c5146 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:07:48.290525 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ee4ae8-dddc-4076-8a3f-14ba9d011347 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:48.298706 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c02a3a45-3848-4ec6-a7bc-2db2a92df7e2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:48.298706 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c02a3a45-3848-4ec6-a7bc-2db2a92df7e2 service cinder] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:48.298706 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c02a3a45-3848-4ec6-a7bc-2db2a92df7e2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:48.298706 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c02a3a45-3848-4ec6-a7bc-2db2a92df7e2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:48.343348 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1104/2202] 10.222.0.36 () {58 vars in 967 bytes} [Thu Oct 20 09:07:48 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:07:48.371020 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20336808-8772-4a71-9ff1-bfee9e4c5146 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:48.372480 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-20336808-8772-4a71-9ff1-bfee9e4c5146 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:07:48.379919 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c02a3a45-3848-4ec6-a7bc-2db2a92df7e2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:48.381498 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c02a3a45-3848-4ec6-a7bc-2db2a92df7e2 service cinder] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:48.394824 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:07:48.401038 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1099/2203] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:07:48 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 201 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:07:48.471429 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20336808-8772-4a71-9ff1-bfee9e4c5146 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:48.480310 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-451f43c6-f96d-46da-a87a-2535085440a0 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4d558918cd8b4ebea4fe8765297304db', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:48.481013 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-451f43c6-f96d-46da-a87a-2535085440a0 service swift] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:48.481362 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-451f43c6-f96d-46da-a87a-2535085440a0 service swift] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:48.481787 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-451f43c6-f96d-46da-a87a-2535085440a0 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:48.491497 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-451f43c6-f96d-46da-a87a-2535085440a0 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:48.493188 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-451f43c6-f96d-46da-a87a-2535085440a0 service swift] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:48.503147 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1105/2204] 10.222.0.36 () {60 vars in 1384 bytes} [Thu Oct 20 09:07:48 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 140 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 20 09:07:54.255187 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c02a3a45-3848-4ec6-a7bc-2db2a92df7e2 service cinder] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:54.256681 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-c02a3a45-3848-4ec6-a7bc-2db2a92df7e2 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:07:54.268322 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c02a3a45-3848-4ec6-a7bc-2db2a92df7e2 service cinder] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:54.272005 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d0c2ea2-c696-4d46-97dd-8b9a6296b74d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:54.272678 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d0c2ea2-c696-4d46-97dd-8b9a6296b74d service cinder] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:54.273023 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d0c2ea2-c696-4d46-97dd-8b9a6296b74d service cinder] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:54.273358 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d0c2ea2-c696-4d46-97dd-8b9a6296b74d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:54.283219 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d0c2ea2-c696-4d46-97dd-8b9a6296b74d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:54.284789 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d0c2ea2-c696-4d46-97dd-8b9a6296b74d service cinder] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:54.296721 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1100/2205] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:07:54 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:07:56.045325 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-451f43c6-f96d-46da-a87a-2535085440a0 service swift] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:56.046033 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-451f43c6-f96d-46da-a87a-2535085440a0 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:07:56.125971 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-451f43c6-f96d-46da-a87a-2535085440a0 service swift] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:56.130503 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8618081-6be8-4477-886b-f7cdbc052db9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:56.131135 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8618081-6be8-4477-886b-f7cdbc052db9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:56.131406 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8618081-6be8-4477-886b-f7cdbc052db9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:56.131669 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8618081-6be8-4477-886b-f7cdbc052db9 admin admin] PATH_INFO: `/v3/users/14ddd15ab87a4d0fbea1d6dfa910f8ce` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:56.132774 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8618081-6be8-4477-886b-f7cdbc052db9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=14ddd15ab87a4d0fbea1d6dfa910f8ce)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:56.134745 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8618081-6be8-4477-886b-f7cdbc052db9 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:56.186155 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-f8618081-6be8-4477-886b-f7cdbc052db9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '14ddd15ab87a4d0fbea1d6dfa910f8ce'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:56.195320 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-f8618081-6be8-4477-886b-f7cdbc052db9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '14ddd15ab87a4d0fbea1d6dfa910f8ce'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:56.209283 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-f8618081-6be8-4477-886b-f7cdbc052db9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '14ddd15ab87a4d0fbea1d6dfa910f8ce'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:56.217441 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-f8618081-6be8-4477-886b-f7cdbc052db9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '14ddd15ab87a4d0fbea1d6dfa910f8ce'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:56.218173 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-f8618081-6be8-4477-886b-f7cdbc052db9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '14ddd15ab87a4d0fbea1d6dfa910f8ce'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:56.224248 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1106/2206] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:07:56 2022] DELETE /identity/v3/users/14ddd15ab87a4d0fbea1d6dfa910f8ce => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:07:56.400764 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d0c2ea2-c696-4d46-97dd-8b9a6296b74d service cinder] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:56.416674 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d0c2ea2-c696-4d46-97dd-8b9a6296b74d service cinder] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:56.451620 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:07:56.475812 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6ba9d47-b1a6-4fed-a809-1664ed3ff91e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:56.476374 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6ba9d47-b1a6-4fed-a809-1664ed3ff91e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:56.476652 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6ba9d47-b1a6-4fed-a809-1664ed3ff91e admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:56.476919 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6ba9d47-b1a6-4fed-a809-1664ed3ff91e admin admin] PATH_INFO: `/v3/projects/020d4019643f4499b5162717fbf7b5b5` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:56.478019 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6ba9d47-b1a6-4fed-a809-1664ed3ff91e admin admin] RBAC: Authorizing `identity:delete_project(project_id=020d4019643f4499b5162717fbf7b5b5)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:56.479969 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6ba9d47-b1a6-4fed-a809-1664ed3ff91e admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:56.525312 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-b6ba9d47-b1a6-4fed-a809-1664ed3ff91e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '020d4019643f4499b5162717fbf7b5b5', 'request_id': 'req-b6ba9d47-b1a6-4fed-a809-1664ed3ff91e'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:56.531404 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-b6ba9d47-b1a6-4fed-a809-1664ed3ff91e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '020d4019643f4499b5162717fbf7b5b5', 'request_id': 'req-b6ba9d47-b1a6-4fed-a809-1664ed3ff91e'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:56.536863 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-b6ba9d47-b1a6-4fed-a809-1664ed3ff91e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '020d4019643f4499b5162717fbf7b5b5', 'request_id': 'req-b6ba9d47-b1a6-4fed-a809-1664ed3ff91e'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:56.549044 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-b6ba9d47-b1a6-4fed-a809-1664ed3ff91e admin admin] The token cache is being invalidate because project 020d4019643f4499b5162717fbf7b5b5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81562) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:07:56.549329 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-b6ba9d47-b1a6-4fed-a809-1664ed3ff91e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:56.549945 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-b6ba9d47-b1a6-4fed-a809-1664ed3ff91e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:56.551441 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1101/2207] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:07:56 2022] DELETE /identity/v3/projects/020d4019643f4499b5162717fbf7b5b5 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:07:56.555582 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8618081-6be8-4477-886b-f7cdbc052db9 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:56.557010 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-f8618081-6be8-4477-886b-f7cdbc052db9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:07:56.570692 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8618081-6be8-4477-886b-f7cdbc052db9 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:56.586729 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:07:56.628540 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-535e4844-8a3f-481b-9fbf-5aa758fe953e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:56.629168 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-535e4844-8a3f-481b-9fbf-5aa758fe953e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:56.629446 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-535e4844-8a3f-481b-9fbf-5aa758fe953e admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:56.629718 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-535e4844-8a3f-481b-9fbf-5aa758fe953e admin admin] PATH_INFO: `/v3/users/69303fcf63304748b1988ce4a19317af` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:56.630864 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-535e4844-8a3f-481b-9fbf-5aa758fe953e admin admin] RBAC: Authorizing `identity:delete_user(user_id=69303fcf63304748b1988ce4a19317af)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:56.632792 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-535e4844-8a3f-481b-9fbf-5aa758fe953e admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:56.678290 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-535e4844-8a3f-481b-9fbf-5aa758fe953e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '69303fcf63304748b1988ce4a19317af'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:56.686800 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-535e4844-8a3f-481b-9fbf-5aa758fe953e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '69303fcf63304748b1988ce4a19317af'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:56.698714 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-535e4844-8a3f-481b-9fbf-5aa758fe953e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '69303fcf63304748b1988ce4a19317af'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:56.705630 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-535e4844-8a3f-481b-9fbf-5aa758fe953e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '69303fcf63304748b1988ce4a19317af'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:56.706276 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-535e4844-8a3f-481b-9fbf-5aa758fe953e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '69303fcf63304748b1988ce4a19317af'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:56.712271 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1107/2208] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:07:56 2022] DELETE /identity/v3/users/69303fcf63304748b1988ce4a19317af => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:07:56.883545 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6ba9d47-b1a6-4fed-a809-1664ed3ff91e admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:07:56.885029 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-b6ba9d47-b1a6-4fed-a809-1664ed3ff91e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:07:56.898465 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6ba9d47-b1a6-4fed-a809-1664ed3ff91e admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:07:56.954209 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1b9e3c0-0ca3-4cec-8bc2-0c13a754f893 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:07:56.954860 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b9e3c0-0ca3-4cec-8bc2-0c13a754f893 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:07:56.955135 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b9e3c0-0ca3-4cec-8bc2-0c13a754f893 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:07:56.955419 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b9e3c0-0ca3-4cec-8bc2-0c13a754f893 admin admin] PATH_INFO: `/v3/projects/7e1c9e1c5a2249008cc4b4c7c12ff61e` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:07:56.956506 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1b9e3c0-0ca3-4cec-8bc2-0c13a754f893 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7e1c9e1c5a2249008cc4b4c7c12ff61e)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:07:56.958616 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1b9e3c0-0ca3-4cec-8bc2-0c13a754f893 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:07:57.003253 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-d1b9e3c0-0ca3-4cec-8bc2-0c13a754f893 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7e1c9e1c5a2249008cc4b4c7c12ff61e', 'request_id': 'req-d1b9e3c0-0ca3-4cec-8bc2-0c13a754f893'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:57.009439 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-d1b9e3c0-0ca3-4cec-8bc2-0c13a754f893 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7e1c9e1c5a2249008cc4b4c7c12ff61e', 'request_id': 'req-d1b9e3c0-0ca3-4cec-8bc2-0c13a754f893'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:57.015340 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-d1b9e3c0-0ca3-4cec-8bc2-0c13a754f893 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7e1c9e1c5a2249008cc4b4c7c12ff61e', 'request_id': 'req-d1b9e3c0-0ca3-4cec-8bc2-0c13a754f893'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:57.023236 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:07:57.029950 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-d1b9e3c0-0ca3-4cec-8bc2-0c13a754f893 admin admin] The token cache is being invalidate because project 7e1c9e1c5a2249008cc4b4c7c12ff61e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81562) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:07:57.030503 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-d1b9e3c0-0ca3-4cec-8bc2-0c13a754f893 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:57.031509 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-d1b9e3c0-0ca3-4cec-8bc2-0c13a754f893 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:07:57.033096 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1102/2209] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:07:56 2022] DELETE /identity/v3/projects/7e1c9e1c5a2249008cc4b4c7c12ff61e => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:08:00.517084 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-535e4844-8a3f-481b-9fbf-5aa758fe953e admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:00.518545 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-535e4844-8a3f-481b-9fbf-5aa758fe953e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:00.580843 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-535e4844-8a3f-481b-9fbf-5aa758fe953e admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:00.635922 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b01a03c-2fe0-4f00-b340-95de4a0fbe3c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:00.636520 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b01a03c-2fe0-4f00-b340-95de4a0fbe3c service neutron] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:00.636782 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b01a03c-2fe0-4f00-b340-95de4a0fbe3c service neutron] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:00.637051 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b01a03c-2fe0-4f00-b340-95de4a0fbe3c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:00.639050 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-7b01a03c-2fe0-4f00-b340-95de4a0fbe3c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:00.648501 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:00.710684 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b01a03c-2fe0-4f00-b340-95de4a0fbe3c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:00.712129 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b01a03c-2fe0-4f00-b340-95de4a0fbe3c service neutron] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:00.722856 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1108/2210] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:08:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 206 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 20 09:08:04.804810 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1b9e3c0-0ca3-4cec-8bc2-0c13a754f893 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:04.806276 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-d1b9e3c0-0ca3-4cec-8bc2-0c13a754f893 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:04.878797 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1b9e3c0-0ca3-4cec-8bc2-0c13a754f893 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:04.883822 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b560eb73-d229-45dc-9e96-72f650e2e6af service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ad03ab76b9c44b1cb369475afa25eddd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:04.884425 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b560eb73-d229-45dc-9e96-72f650e2e6af service placement] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:04.884702 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b560eb73-d229-45dc-9e96-72f650e2e6af service placement] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:04.884976 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b560eb73-d229-45dc-9e96-72f650e2e6af service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:04.887060 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-b560eb73-d229-45dc-9e96-72f650e2e6af service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:04.901865 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b01a03c-2fe0-4f00-b340-95de4a0fbe3c service neutron] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:04.904283 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-7b01a03c-2fe0-4f00-b340-95de4a0fbe3c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:04.906996 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b560eb73-d229-45dc-9e96-72f650e2e6af service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:04.909044 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b560eb73-d229-45dc-9e96-72f650e2e6af service placement] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:04.923186 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1103/2211] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:08:04 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:08:04.923787 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b01a03c-2fe0-4f00-b340-95de4a0fbe3c service neutron] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:04.930561 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-934f7ec0-b426-4351-81eb-4bfe464b45b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:04.931336 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-934f7ec0-b426-4351-81eb-4bfe464b45b8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:04.931737 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-934f7ec0-b426-4351-81eb-4bfe464b45b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:04.932073 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-934f7ec0-b426-4351-81eb-4bfe464b45b8 admin admin] PATH_INFO: `/v3/users/be1f82d7a0a84fbebdd0dffef9f098ea` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:04.933654 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-934f7ec0-b426-4351-81eb-4bfe464b45b8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=be1f82d7a0a84fbebdd0dffef9f098ea)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:04.936206 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-934f7ec0-b426-4351-81eb-4bfe464b45b8 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:05.004335 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-934f7ec0-b426-4351-81eb-4bfe464b45b8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'be1f82d7a0a84fbebdd0dffef9f098ea'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:05.017027 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-934f7ec0-b426-4351-81eb-4bfe464b45b8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'be1f82d7a0a84fbebdd0dffef9f098ea'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:05.035392 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-934f7ec0-b426-4351-81eb-4bfe464b45b8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'be1f82d7a0a84fbebdd0dffef9f098ea'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:05.043543 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-934f7ec0-b426-4351-81eb-4bfe464b45b8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'be1f82d7a0a84fbebdd0dffef9f098ea'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:05.044973 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-934f7ec0-b426-4351-81eb-4bfe464b45b8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'be1f82d7a0a84fbebdd0dffef9f098ea'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:05.058119 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1109/2212] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:08:04 2022] DELETE /identity/v3/users/be1f82d7a0a84fbebdd0dffef9f098ea => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:08:05.301828 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b560eb73-d229-45dc-9e96-72f650e2e6af service placement] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:05.320548 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b560eb73-d229-45dc-9e96-72f650e2e6af service placement] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:05.403134 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55917058-c872-42fe-9935-e31172d62d55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:05.403817 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55917058-c872-42fe-9935-e31172d62d55 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:05.404112 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55917058-c872-42fe-9935-e31172d62d55 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:05.404372 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55917058-c872-42fe-9935-e31172d62d55 admin admin] PATH_INFO: `/v3/projects/f1061324903c42ed9f0306471a113b82` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:05.407021 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55917058-c872-42fe-9935-e31172d62d55 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f1061324903c42ed9f0306471a113b82)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:05.409064 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55917058-c872-42fe-9935-e31172d62d55 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:05.472895 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-55917058-c872-42fe-9935-e31172d62d55 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f1061324903c42ed9f0306471a113b82', 'request_id': 'req-55917058-c872-42fe-9935-e31172d62d55'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:05.480852 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-55917058-c872-42fe-9935-e31172d62d55 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f1061324903c42ed9f0306471a113b82', 'request_id': 'req-55917058-c872-42fe-9935-e31172d62d55'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:05.487795 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-55917058-c872-42fe-9935-e31172d62d55 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f1061324903c42ed9f0306471a113b82', 'request_id': 'req-55917058-c872-42fe-9935-e31172d62d55'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:05.501137 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-55917058-c872-42fe-9935-e31172d62d55 admin admin] The token cache is being invalidate because project f1061324903c42ed9f0306471a113b82 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81562) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:08:05.501444 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-55917058-c872-42fe-9935-e31172d62d55 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:05.502052 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-55917058-c872-42fe-9935-e31172d62d55 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:05.503576 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1104/2213] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:08:05 2022] DELETE /identity/v3/projects/f1061324903c42ed9f0306471a113b82 => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:08:05.508696 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a38b06-c44c-4692-8508-fa61fbc285cd None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:05.509075 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a38b06-c44c-4692-8508-fa61fbc285cd None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:05.509480 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a38b06-c44c-4692-8508-fa61fbc285cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:05.530516 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-a7a38b06-c44c-4692-8508-fa61fbc285cd None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:08:05.531651 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-a7a38b06-c44c-4692-8508-fa61fbc285cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:05.599018 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1110/2214] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:08:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:08:05.607550 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80930583-7c4b-4962-bdb9-cec12c9ab82f None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:05.607843 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80930583-7c4b-4962-bdb9-cec12c9ab82f None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:05.608119 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80930583-7c4b-4962-bdb9-cec12c9ab82f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:05.628616 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-80930583-7c4b-4962-bdb9-cec12c9ab82f None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:08:05.629848 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-80930583-7c4b-4962-bdb9-cec12c9ab82f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:05.638803 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1105/2215] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:08:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:08:05.643143 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7a38b06-c44c-4692-8508-fa61fbc285cd None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:05.654534 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7a38b06-c44c-4692-8508-fa61fbc285cd None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:05.660201 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c587ec-f3f7-4511-9967-a4ee9a5b1bc0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:05.660906 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c587ec-f3f7-4511-9967-a4ee9a5b1bc0 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:05.661250 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c587ec-f3f7-4511-9967-a4ee9a5b1bc0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:05.661585 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c587ec-f3f7-4511-9967-a4ee9a5b1bc0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:05.662364 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48c587ec-f3f7-4511-9967-a4ee9a5b1bc0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 09:08:05.662698 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48c587ec-f3f7-4511-9967-a4ee9a5b1bc0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:05.664317 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48c587ec-f3f7-4511-9967-a4ee9a5b1bc0 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:05.680710 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1111/2216] 10.222.0.36 () {58 vars in 1153 bytes} [Thu Oct 20 09:08:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 09:08:05.682429 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80930583-7c4b-4962-bdb9-cec12c9ab82f None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:05.691881 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80930583-7c4b-4962-bdb9-cec12c9ab82f None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:05.698915 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be94fec-3fdb-4835-9f68-877931ecf3be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:05.699533 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be94fec-3fdb-4835-9f68-877931ecf3be admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:05.700419 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be94fec-3fdb-4835-9f68-877931ecf3be admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:05.700634 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be94fec-3fdb-4835-9f68-877931ecf3be admin admin] PATH_INFO: `/v3/projects` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:05.701272 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8be94fec-3fdb-4835-9f68-877931ecf3be admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:05.703231 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8be94fec-3fdb-4835-9f68-877931ecf3be admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:05.725560 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:05.747150 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1106/2217] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:08:05 2022] POST /identity/v3/projects => generated 405 bytes in 64 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:08:05.750853 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c587ec-f3f7-4511-9967-a4ee9a5b1bc0 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:05.758000 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:05.763756 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c587ec-f3f7-4511-9967-a4ee9a5b1bc0 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:05.849208 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51ba5c05-031f-400e-8684-6abe987b84e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:05.850020 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ba5c05-031f-400e-8684-6abe987b84e9 admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:05.850177 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ba5c05-031f-400e-8684-6abe987b84e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:05.850508 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ba5c05-031f-400e-8684-6abe987b84e9 admin admin] PATH_INFO: `/v3/users` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:05.851549 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51ba5c05-031f-400e-8684-6abe987b84e9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:05.853589 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51ba5c05-031f-400e-8684-6abe987b84e9 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:05.882525 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1112/2218] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:08:05 2022] POST /identity/v3/users => generated 303 bytes in 131 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:08:05.885319 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be94fec-3fdb-4835-9f68-877931ecf3be admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:05.897252 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be94fec-3fdb-4835-9f68-877931ecf3be admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:05.902752 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec543f35-8f17-438c-8b00-381a21e27ecb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:05.904014 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec543f35-8f17-438c-8b00-381a21e27ecb admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:05.904298 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec543f35-8f17-438c-8b00-381a21e27ecb admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:05.904562 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec543f35-8f17-438c-8b00-381a21e27ecb admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:05.905705 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec543f35-8f17-438c-8b00-381a21e27ecb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:05.907353 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec543f35-8f17-438c-8b00-381a21e27ecb admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:05.920437 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1107/2219] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:08:05 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:08:05.924184 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51ba5c05-031f-400e-8684-6abe987b84e9 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:05.935866 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51ba5c05-031f-400e-8684-6abe987b84e9 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:05.941439 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7917c5df-8d4d-4ca1-a4a2-d80f03e7f619 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:05.942082 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7917c5df-8d4d-4ca1-a4a2-d80f03e7f619 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:05.942840 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7917c5df-8d4d-4ca1-a4a2-d80f03e7f619 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:05.943315 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7917c5df-8d4d-4ca1-a4a2-d80f03e7f619 admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:05.944342 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7917c5df-8d4d-4ca1-a4a2-d80f03e7f619 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:05.948790 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7917c5df-8d4d-4ca1-a4a2-d80f03e7f619 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:05.962613 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1113/2220] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:08:05 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:08:05.964533 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec543f35-8f17-438c-8b00-381a21e27ecb admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:05.977341 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec543f35-8f17-438c-8b00-381a21e27ecb admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:05.982810 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a5017ee-377b-4fa0-a0f5-da3302386d0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:05.983478 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a5017ee-377b-4fa0-a0f5-da3302386d0b admin admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:05.983751 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a5017ee-377b-4fa0-a0f5-da3302386d0b admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:05.984015 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a5017ee-377b-4fa0-a0f5-da3302386d0b admin admin] PATH_INFO: `/v3/projects/f8a54576c7d647bb9fb58a93f62e8aab/users/119e3c15a4ef488cabbb80f9ddb2ca31/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:05.999214 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a5017ee-377b-4fa0-a0f5-da3302386d0b admin admin] RBAC: Authorizing `identity:create_grant(project_id=f8a54576c7d647bb9fb58a93f62e8aab, user_id=119e3c15a4ef488cabbb80f9ddb2ca31, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:06.001863 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a5017ee-377b-4fa0-a0f5-da3302386d0b admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:06.015206 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1108/2221] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:08:05 2022] PUT /identity/v3/projects/f8a54576c7d647bb9fb58a93f62e8aab/users/119e3c15a4ef488cabbb80f9ddb2ca31/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:08:06.020530 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75464da3-1d04-4ce2-adec-3de4fc430db0 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:06.020815 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75464da3-1d04-4ce2-adec-3de4fc430db0 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:06.021082 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75464da3-1d04-4ce2-adec-3de4fc430db0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:06.040637 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-75464da3-1d04-4ce2-adec-3de4fc430db0 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:08:06.041833 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-75464da3-1d04-4ce2-adec-3de4fc430db0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:06.114096 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1114/2222] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:08:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 20 09:08:06.128903 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a5017ee-377b-4fa0-a0f5-da3302386d0b admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:06.131065 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-9a5017ee-377b-4fa0-a0f5-da3302386d0b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:06.211398 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a5017ee-377b-4fa0-a0f5-da3302386d0b admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:06.216529 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-731cd2f8-9514-4fd2-a3d3-532d6af92a41 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:06.217211 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731cd2f8-9514-4fd2-a3d3-532d6af92a41 service neutron] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:06.217552 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731cd2f8-9514-4fd2-a3d3-532d6af92a41 service neutron] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:06.217886 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731cd2f8-9514-4fd2-a3d3-532d6af92a41 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:06.227366 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-731cd2f8-9514-4fd2-a3d3-532d6af92a41 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:06.228881 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-731cd2f8-9514-4fd2-a3d3-532d6af92a41 service neutron] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:06.244666 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1109/2223] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:08:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:08:06.932104 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-549f7761-c96f-4764-b572-5fd010ed490f None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:06.932655 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-549f7761-c96f-4764-b572-5fd010ed490f None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:06.932655 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-549f7761-c96f-4764-b572-5fd010ed490f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:06.961440 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-549f7761-c96f-4764-b572-5fd010ed490f None None] MFA Rules not processed for user `119e3c15a4ef488cabbb80f9ddb2ca31`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:08:06.963196 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-549f7761-c96f-4764-b572-5fd010ed490f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:07.031136 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1115/2224] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:08:06 2022] POST /identity/v3/auth/tokens => generated 3282 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:08:07.035609 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-731cd2f8-9514-4fd2-a3d3-532d6af92a41 service neutron] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:07.046422 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-731cd2f8-9514-4fd2-a3d3-532d6af92a41 service neutron] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:07.051175 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19dbd411-c4c6-4b16-b528-75ad3df595e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:07.051788 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19dbd411-c4c6-4b16-b528-75ad3df595e5 admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:07.052089 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19dbd411-c4c6-4b16-b528-75ad3df595e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:07.052364 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19dbd411-c4c6-4b16-b528-75ad3df595e5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:07.053005 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19dbd411-c4c6-4b16-b528-75ad3df595e5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:07.055018 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19dbd411-c4c6-4b16-b528-75ad3df595e5 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:07.084365 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1110/2225] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:08:07 2022] POST /identity/v3/projects => generated 401 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:08:07.088558 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-549f7761-c96f-4764-b572-5fd010ed490f None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:07.100164 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-549f7761-c96f-4764-b572-5fd010ed490f None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:07.110323 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:07.162942 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2d35134-efdd-4bbb-bffa-a1cca4851e29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:07.163571 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d35134-efdd-4bbb-bffa-a1cca4851e29 admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:07.165950 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d35134-efdd-4bbb-bffa-a1cca4851e29 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:07.165950 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d35134-efdd-4bbb-bffa-a1cca4851e29 admin admin] PATH_INFO: `/v3/users` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:07.165950 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2d35134-efdd-4bbb-bffa-a1cca4851e29 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:07.166978 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2d35134-efdd-4bbb-bffa-a1cca4851e29 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:07.194431 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1116/2226] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:08:07 2022] POST /identity/v3/users => generated 307 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:08:07.196649 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19dbd411-c4c6-4b16-b528-75ad3df595e5 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:07.212379 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19dbd411-c4c6-4b16-b528-75ad3df595e5 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:07.217778 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61e47a80-cb0b-4bf6-872a-0eb3003fd5c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:07.218381 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e47a80-cb0b-4bf6-872a-0eb3003fd5c0 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:07.218692 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e47a80-cb0b-4bf6-872a-0eb3003fd5c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:07.218988 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e47a80-cb0b-4bf6-872a-0eb3003fd5c0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:07.219630 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61e47a80-cb0b-4bf6-872a-0eb3003fd5c0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:07.221236 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61e47a80-cb0b-4bf6-872a-0eb3003fd5c0 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:07.233256 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1111/2227] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:08:07 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:08:07.237382 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2d35134-efdd-4bbb-bffa-a1cca4851e29 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:07.248517 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2d35134-efdd-4bbb-bffa-a1cca4851e29 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:07.253118 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93053194-b610-4085-be50-41bd54c7c762 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:07.253768 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93053194-b610-4085-be50-41bd54c7c762 admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:07.254042 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93053194-b610-4085-be50-41bd54c7c762 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:07.254309 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93053194-b610-4085-be50-41bd54c7c762 admin admin] PATH_INFO: `/v3/projects/e2a94a401006415499f33a68ba88e97d/users/de1a7a9a8d0f4b368c764b7a3186227e/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:07.269329 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93053194-b610-4085-be50-41bd54c7c762 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e2a94a401006415499f33a68ba88e97d, user_id=de1a7a9a8d0f4b368c764b7a3186227e, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:07.272021 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93053194-b610-4085-be50-41bd54c7c762 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:07.284770 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1117/2228] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:08:07 2022] PUT /identity/v3/projects/e2a94a401006415499f33a68ba88e97d/users/de1a7a9a8d0f4b368c764b7a3186227e/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:08:07.288862 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61e47a80-cb0b-4bf6-872a-0eb3003fd5c0 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:07.299827 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61e47a80-cb0b-4bf6-872a-0eb3003fd5c0 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:07.357776 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a3bed97-c602-4378-9d72-2377136c4e75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:07.358388 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a3bed97-c602-4378-9d72-2377136c4e75 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:07.358704 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a3bed97-c602-4378-9d72-2377136c4e75 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:07.358972 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a3bed97-c602-4378-9d72-2377136c4e75 admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:07.359623 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a3bed97-c602-4378-9d72-2377136c4e75 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:07.361214 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a3bed97-c602-4378-9d72-2377136c4e75 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:07.377267 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1112/2229] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:08:07 2022] GET /identity/v3/roles => generated 1420 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:08:07.381528 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93053194-b610-4085-be50-41bd54c7c762 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:07.392341 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93053194-b610-4085-be50-41bd54c7c762 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:07.396928 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c1a6a00-833f-4303-8349-9f2f91198246 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:07.397568 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c1a6a00-833f-4303-8349-9f2f91198246 admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:07.397840 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c1a6a00-833f-4303-8349-9f2f91198246 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:07.398102 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c1a6a00-833f-4303-8349-9f2f91198246 admin admin] PATH_INFO: `/v3/projects/e2a94a401006415499f33a68ba88e97d/users/de1a7a9a8d0f4b368c764b7a3186227e/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:07.400088 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c1a6a00-833f-4303-8349-9f2f91198246 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e2a94a401006415499f33a68ba88e97d, user_id=de1a7a9a8d0f4b368c764b7a3186227e, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:07.402754 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c1a6a00-833f-4303-8349-9f2f91198246 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:07.413633 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1118/2230] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:08:07 2022] PUT /identity/v3/projects/e2a94a401006415499f33a68ba88e97d/users/de1a7a9a8d0f4b368c764b7a3186227e/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:08:08.108574 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185191c2-73b1-4b57-bcc4-55ebeaf00132 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:08.109098 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185191c2-73b1-4b57-bcc4-55ebeaf00132 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:08.109542 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185191c2-73b1-4b57-bcc4-55ebeaf00132 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:08.138351 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-185191c2-73b1-4b57-bcc4-55ebeaf00132 None None] MFA Rules not processed for user `de1a7a9a8d0f4b368c764b7a3186227e`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:08:08.139758 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-185191c2-73b1-4b57-bcc4-55ebeaf00132 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:08.148057 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:08.219299 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1113/2231] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:08:08 2022] POST /identity/v3/auth/tokens => generated 3345 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:08:09.242617 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c1a6a00-833f-4303-8349-9f2f91198246 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:09.306937 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c1a6a00-833f-4303-8349-9f2f91198246 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:09.312101 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-549f13e8-d51b-4c1b-8287-e45f251cf3d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:09.312720 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-549f13e8-d51b-4c1b-8287-e45f251cf3d5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:09.312996 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-549f13e8-d51b-4c1b-8287-e45f251cf3d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:09.313265 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-549f13e8-d51b-4c1b-8287-e45f251cf3d5 admin admin] PATH_INFO: `/v3/users/119e3c15a4ef488cabbb80f9ddb2ca31` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:09.314368 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-549f13e8-d51b-4c1b-8287-e45f251cf3d5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=119e3c15a4ef488cabbb80f9ddb2ca31)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:09.316316 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-549f13e8-d51b-4c1b-8287-e45f251cf3d5 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:09.362961 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-549f13e8-d51b-4c1b-8287-e45f251cf3d5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '119e3c15a4ef488cabbb80f9ddb2ca31'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:09.373403 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-549f13e8-d51b-4c1b-8287-e45f251cf3d5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '119e3c15a4ef488cabbb80f9ddb2ca31'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:09.388824 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=85,keystone:INSERT=6,keystone:DELETE=9 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:09.391290 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-549f13e8-d51b-4c1b-8287-e45f251cf3d5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '119e3c15a4ef488cabbb80f9ddb2ca31'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:09.400267 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-549f13e8-d51b-4c1b-8287-e45f251cf3d5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '119e3c15a4ef488cabbb80f9ddb2ca31'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:09.400982 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-549f13e8-d51b-4c1b-8287-e45f251cf3d5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '119e3c15a4ef488cabbb80f9ddb2ca31'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:09.407293 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1119/2232] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:08:09 2022] DELETE /identity/v3/users/119e3c15a4ef488cabbb80f9ddb2ca31 => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:08:09.584497 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-185191c2-73b1-4b57-bcc4-55ebeaf00132 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:09.595596 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-185191c2-73b1-4b57-bcc4-55ebeaf00132 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:09.654131 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23f57124-d31f-4754-945a-e64a2bae655a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:09.654893 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f57124-d31f-4754-945a-e64a2bae655a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:09.655234 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f57124-d31f-4754-945a-e64a2bae655a admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:09.655652 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f57124-d31f-4754-945a-e64a2bae655a admin admin] PATH_INFO: `/v3/projects/f8a54576c7d647bb9fb58a93f62e8aab` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:09.656866 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23f57124-d31f-4754-945a-e64a2bae655a admin admin] RBAC: Authorizing `identity:delete_project(project_id=f8a54576c7d647bb9fb58a93f62e8aab)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:09.658950 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23f57124-d31f-4754-945a-e64a2bae655a admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:09.714744 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-23f57124-d31f-4754-945a-e64a2bae655a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f8a54576c7d647bb9fb58a93f62e8aab', 'request_id': 'req-23f57124-d31f-4754-945a-e64a2bae655a'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:09.721459 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-23f57124-d31f-4754-945a-e64a2bae655a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f8a54576c7d647bb9fb58a93f62e8aab', 'request_id': 'req-23f57124-d31f-4754-945a-e64a2bae655a'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:09.729319 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-23f57124-d31f-4754-945a-e64a2bae655a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f8a54576c7d647bb9fb58a93f62e8aab', 'request_id': 'req-23f57124-d31f-4754-945a-e64a2bae655a'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:09.743977 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-23f57124-d31f-4754-945a-e64a2bae655a admin admin] The token cache is being invalidate because project f8a54576c7d647bb9fb58a93f62e8aab was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81562) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:08:09.744261 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-23f57124-d31f-4754-945a-e64a2bae655a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:09.744898 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-23f57124-d31f-4754-945a-e64a2bae655a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:09.747254 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1114/2233] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:08:09 2022] DELETE /identity/v3/projects/f8a54576c7d647bb9fb58a93f62e8aab => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:08:09.751998 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-549f13e8-d51b-4c1b-8287-e45f251cf3d5 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:09.753585 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-549f13e8-d51b-4c1b-8287-e45f251cf3d5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:09.767587 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-549f13e8-d51b-4c1b-8287-e45f251cf3d5 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:09.826672 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82bcee07-93f0-440a-bb5b-d754bda97c98 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:09.827309 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82bcee07-93f0-440a-bb5b-d754bda97c98 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:09.827584 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82bcee07-93f0-440a-bb5b-d754bda97c98 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:09.827861 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82bcee07-93f0-440a-bb5b-d754bda97c98 admin admin] PATH_INFO: `/v3/users/de1a7a9a8d0f4b368c764b7a3186227e` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:09.828997 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82bcee07-93f0-440a-bb5b-d754bda97c98 admin admin] RBAC: Authorizing `identity:delete_user(user_id=de1a7a9a8d0f4b368c764b7a3186227e)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:09.830973 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82bcee07-93f0-440a-bb5b-d754bda97c98 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:09.881896 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-82bcee07-93f0-440a-bb5b-d754bda97c98 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'de1a7a9a8d0f4b368c764b7a3186227e'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:09.891599 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-82bcee07-93f0-440a-bb5b-d754bda97c98 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'de1a7a9a8d0f4b368c764b7a3186227e'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:09.909937 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-82bcee07-93f0-440a-bb5b-d754bda97c98 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'de1a7a9a8d0f4b368c764b7a3186227e'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:09.916900 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-82bcee07-93f0-440a-bb5b-d754bda97c98 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'de1a7a9a8d0f4b368c764b7a3186227e'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:09.917604 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-82bcee07-93f0-440a-bb5b-d754bda97c98 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'de1a7a9a8d0f4b368c764b7a3186227e'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:09.923502 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1120/2234] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:08:09 2022] DELETE /identity/v3/users/de1a7a9a8d0f4b368c764b7a3186227e => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:08:10.096383 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23f57124-d31f-4754-945a-e64a2bae655a admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:10.097830 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-23f57124-d31f-4754-945a-e64a2bae655a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:10.111120 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23f57124-d31f-4754-945a-e64a2bae655a admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:10.148336 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:10.174383 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd8fb808-0597-4ace-baff-eaf3252d8f19 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:10.175044 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd8fb808-0597-4ace-baff-eaf3252d8f19 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:10.175326 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd8fb808-0597-4ace-baff-eaf3252d8f19 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:10.175602 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd8fb808-0597-4ace-baff-eaf3252d8f19 admin admin] PATH_INFO: `/v3/projects/e2a94a401006415499f33a68ba88e97d` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:10.176731 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd8fb808-0597-4ace-baff-eaf3252d8f19 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e2a94a401006415499f33a68ba88e97d)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:10.178676 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd8fb808-0597-4ace-baff-eaf3252d8f19 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:10.232702 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-bd8fb808-0597-4ace-baff-eaf3252d8f19 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e2a94a401006415499f33a68ba88e97d', 'request_id': 'req-bd8fb808-0597-4ace-baff-eaf3252d8f19'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:10.239366 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-bd8fb808-0597-4ace-baff-eaf3252d8f19 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e2a94a401006415499f33a68ba88e97d', 'request_id': 'req-bd8fb808-0597-4ace-baff-eaf3252d8f19'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:10.245317 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-bd8fb808-0597-4ace-baff-eaf3252d8f19 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e2a94a401006415499f33a68ba88e97d', 'request_id': 'req-bd8fb808-0597-4ace-baff-eaf3252d8f19'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:10.258374 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-bd8fb808-0597-4ace-baff-eaf3252d8f19 admin admin] The token cache is being invalidate because project e2a94a401006415499f33a68ba88e97d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81562) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:08:10.258699 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-bd8fb808-0597-4ace-baff-eaf3252d8f19 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:10.259326 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-bd8fb808-0597-4ace-baff-eaf3252d8f19 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:10.260800 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1115/2235] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:08:10 2022] DELETE /identity/v3/projects/e2a94a401006415499f33a68ba88e97d => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:08:10.265712 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184d5f49-aef7-4b19-8ab4-025f3012d047 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:10.266191 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184d5f49-aef7-4b19-8ab4-025f3012d047 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:10.266436 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184d5f49-aef7-4b19-8ab4-025f3012d047 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:10.286452 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-184d5f49-aef7-4b19-8ab4-025f3012d047 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:08:10.287662 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-184d5f49-aef7-4b19-8ab4-025f3012d047 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:10.347802 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1121/2236] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:08:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:08:10.355522 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b8c82b-c5c2-42ea-9326-1451f21e6331 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:10.356493 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b8c82b-c5c2-42ea-9326-1451f21e6331 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:10.356807 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b8c82b-c5c2-42ea-9326-1451f21e6331 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:10.381979 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-89b8c82b-c5c2-42ea-9326-1451f21e6331 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:08:10.383222 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-89b8c82b-c5c2-42ea-9326-1451f21e6331 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:10.391468 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1116/2237] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:08:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:08:10.396023 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-184d5f49-aef7-4b19-8ab4-025f3012d047 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:10.403327 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:DELETE=11,keystone:SELECT=88,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:10.407264 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-184d5f49-aef7-4b19-8ab4-025f3012d047 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:10.413421 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c018660e-e2ae-49b4-a26a-780ec9a28e09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:10.414290 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c018660e-e2ae-49b4-a26a-780ec9a28e09 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:10.414619 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c018660e-e2ae-49b4-a26a-780ec9a28e09 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:10.415055 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c018660e-e2ae-49b4-a26a-780ec9a28e09 admin admin] PATH_INFO: `/v3/domains` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:10.415754 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c018660e-e2ae-49b4-a26a-780ec9a28e09 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 09:08:10.416209 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c018660e-e2ae-49b4-a26a-780ec9a28e09 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:10.417821 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c018660e-e2ae-49b4-a26a-780ec9a28e09 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:10.430546 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1122/2238] 10.222.0.36 () {58 vars in 1153 bytes} [Thu Oct 20 09:08:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 09:08:10.434833 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b8c82b-c5c2-42ea-9326-1451f21e6331 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:10.444034 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b8c82b-c5c2-42ea-9326-1451f21e6331 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:10.449344 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c251111-743a-4701-96e0-1936e325533f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:10.449940 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c251111-743a-4701-96e0-1936e325533f admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:10.450217 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c251111-743a-4701-96e0-1936e325533f admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:10.450547 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c251111-743a-4701-96e0-1936e325533f admin admin] PATH_INFO: `/v3/projects` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:10.451155 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c251111-743a-4701-96e0-1936e325533f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:10.453068 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c251111-743a-4701-96e0-1936e325533f admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:10.481785 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1117/2239] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:08:10 2022] POST /identity/v3/projects => generated 371 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:08:10.486002 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c018660e-e2ae-49b4-a26a-780ec9a28e09 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:10.495850 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c018660e-e2ae-49b4-a26a-780ec9a28e09 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:10.550966 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17f7df6a-29ec-4048-afa0-16b8d1fc7922 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:10.551563 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f7df6a-29ec-4048-afa0-16b8d1fc7922 admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:10.551841 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f7df6a-29ec-4048-afa0-16b8d1fc7922 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:10.552109 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f7df6a-29ec-4048-afa0-16b8d1fc7922 admin admin] PATH_INFO: `/v3/users` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:10.552719 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17f7df6a-29ec-4048-afa0-16b8d1fc7922 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:10.554657 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17f7df6a-29ec-4048-afa0-16b8d1fc7922 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:10.575866 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1123/2240] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:08:10 2022] POST /identity/v3/users => generated 286 bytes in 90 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:08:10.579883 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c251111-743a-4701-96e0-1936e325533f admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:10.590261 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c251111-743a-4701-96e0-1936e325533f admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:10.594841 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f543b818-dbfb-4312-8df3-b2a384346ef1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:10.595425 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f543b818-dbfb-4312-8df3-b2a384346ef1 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:10.595702 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f543b818-dbfb-4312-8df3-b2a384346ef1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:10.595970 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f543b818-dbfb-4312-8df3-b2a384346ef1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:10.596631 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f543b818-dbfb-4312-8df3-b2a384346ef1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:10.598210 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f543b818-dbfb-4312-8df3-b2a384346ef1 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:10.608548 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1118/2241] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:08:10 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:08:10.612393 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17f7df6a-29ec-4048-afa0-16b8d1fc7922 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:10.625086 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17f7df6a-29ec-4048-afa0-16b8d1fc7922 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:10.630173 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5230395-0243-4a86-9a62-76518ae108a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:10.630821 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5230395-0243-4a86-9a62-76518ae108a4 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:10.631102 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5230395-0243-4a86-9a62-76518ae108a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:10.631404 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5230395-0243-4a86-9a62-76518ae108a4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:10.632022 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5230395-0243-4a86-9a62-76518ae108a4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:10.633624 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5230395-0243-4a86-9a62-76518ae108a4 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:10.644508 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1124/2242] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:08:10 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:08:10.648673 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f543b818-dbfb-4312-8df3-b2a384346ef1 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:10.658690 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f543b818-dbfb-4312-8df3-b2a384346ef1 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:10.663230 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-503af6d4-3b08-4574-8d16-001db4297614 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:10.663862 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-503af6d4-3b08-4574-8d16-001db4297614 admin admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:10.664139 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-503af6d4-3b08-4574-8d16-001db4297614 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:10.664409 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-503af6d4-3b08-4574-8d16-001db4297614 admin admin] PATH_INFO: `/v3/projects/d5af6df258994fc194d696afb5b384a8/users/2a92f39668bd4de3ba40b34edac35759/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:10.679574 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-503af6d4-3b08-4574-8d16-001db4297614 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d5af6df258994fc194d696afb5b384a8, user_id=2a92f39668bd4de3ba40b34edac35759, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:10.682261 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-503af6d4-3b08-4574-8d16-001db4297614 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:10.696751 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1119/2243] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:08:10 2022] PUT /identity/v3/projects/d5af6df258994fc194d696afb5b384a8/users/2a92f39668bd4de3ba40b34edac35759/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:08:10.707408 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-053e7ab2-9c56-4493-bc31-b91b15ba8d55 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:10.707694 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-053e7ab2-9c56-4493-bc31-b91b15ba8d55 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:10.707969 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-053e7ab2-9c56-4493-bc31-b91b15ba8d55 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:10.736493 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-053e7ab2-9c56-4493-bc31-b91b15ba8d55 None None] MFA Rules not processed for user `2a92f39668bd4de3ba40b34edac35759`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:08:10.737687 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-053e7ab2-9c56-4493-bc31-b91b15ba8d55 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:10.800311 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1125/2244] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:08:10 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:08:10.804784 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-503af6d4-3b08-4574-8d16-001db4297614 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:10.865309 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-503af6d4-3b08-4574-8d16-001db4297614 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:10.869917 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f08f61ad-b92d-4908-bc39-e712012574bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:10.870530 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f08f61ad-b92d-4908-bc39-e712012574bf admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:10.870813 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f08f61ad-b92d-4908-bc39-e712012574bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:10.871084 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f08f61ad-b92d-4908-bc39-e712012574bf admin admin] PATH_INFO: `/v3/projects` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:10.872329 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f08f61ad-b92d-4908-bc39-e712012574bf admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:10.874233 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f08f61ad-b92d-4908-bc39-e712012574bf admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:10.879490 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:10.904313 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1120/2245] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:08:10 2022] POST /identity/v3/projects => generated 371 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:08:10.908555 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-053e7ab2-9c56-4493-bc31-b91b15ba8d55 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:10.919249 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-053e7ab2-9c56-4493-bc31-b91b15ba8d55 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:10.978847 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f4f6d4-7035-4bb4-8fda-527c35882802 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:10.979461 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f4f6d4-7035-4bb4-8fda-527c35882802 admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:10.979740 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f4f6d4-7035-4bb4-8fda-527c35882802 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:10.980009 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f4f6d4-7035-4bb4-8fda-527c35882802 admin admin] PATH_INFO: `/v3/users` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:10.980639 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05f4f6d4-7035-4bb4-8fda-527c35882802 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:10.983177 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05f4f6d4-7035-4bb4-8fda-527c35882802 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:10.991292 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:11.007864 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1126/2246] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:08:10 2022] POST /identity/v3/users => generated 292 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 20 09:08:11.012891 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f08f61ad-b92d-4908-bc39-e712012574bf admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:11.023362 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f08f61ad-b92d-4908-bc39-e712012574bf admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:11.028674 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20e7b5e3-36a1-4d90-aba1-282b27b34de0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:11.029330 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e7b5e3-36a1-4d90-aba1-282b27b34de0 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:11.029671 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e7b5e3-36a1-4d90-aba1-282b27b34de0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:11.030383 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e7b5e3-36a1-4d90-aba1-282b27b34de0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:11.030720 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20e7b5e3-36a1-4d90-aba1-282b27b34de0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:11.032396 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20e7b5e3-36a1-4d90-aba1-282b27b34de0 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:11.043228 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1121/2247] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:08:11 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:08:11.047383 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f4f6d4-7035-4bb4-8fda-527c35882802 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:11.057491 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f4f6d4-7035-4bb4-8fda-527c35882802 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:11.062466 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9c85888-0f4d-481b-8196-354e5c700be2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:11.063284 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c85888-0f4d-481b-8196-354e5c700be2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:11.063560 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c85888-0f4d-481b-8196-354e5c700be2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:11.063824 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c85888-0f4d-481b-8196-354e5c700be2 admin admin] PATH_INFO: `/v3/projects/a137d35b3eed42afb8f51f6167752075/users/1cbc90af37634ccbbd1a1b7bf7b14fd5/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:11.080352 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9c85888-0f4d-481b-8196-354e5c700be2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a137d35b3eed42afb8f51f6167752075, user_id=1cbc90af37634ccbbd1a1b7bf7b14fd5, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:11.083106 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9c85888-0f4d-481b-8196-354e5c700be2 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:11.096321 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1127/2248] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:08:11 2022] PUT /identity/v3/projects/a137d35b3eed42afb8f51f6167752075/users/1cbc90af37634ccbbd1a1b7bf7b14fd5/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:08:11.100228 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20e7b5e3-36a1-4d90-aba1-282b27b34de0 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:11.110762 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20e7b5e3-36a1-4d90-aba1-282b27b34de0 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:11.168294 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25c5fab8-d512-4679-b0ac-6b22ec9e274a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:11.169405 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c5fab8-d512-4679-b0ac-6b22ec9e274a admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:11.169679 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c5fab8-d512-4679-b0ac-6b22ec9e274a admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:11.169946 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c5fab8-d512-4679-b0ac-6b22ec9e274a admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:11.170607 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25c5fab8-d512-4679-b0ac-6b22ec9e274a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:11.172203 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25c5fab8-d512-4679-b0ac-6b22ec9e274a admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:11.181782 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1122/2249] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:08:11 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:08:11.186485 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9c85888-0f4d-481b-8196-354e5c700be2 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:11.197672 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9c85888-0f4d-481b-8196-354e5c700be2 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:11.202305 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f28f6e1-88de-4726-8db1-5bdbb4a03c7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:11.202995 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f28f6e1-88de-4726-8db1-5bdbb4a03c7e admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:11.203274 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f28f6e1-88de-4726-8db1-5bdbb4a03c7e admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:11.203547 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f28f6e1-88de-4726-8db1-5bdbb4a03c7e admin admin] PATH_INFO: `/v3/projects/a137d35b3eed42afb8f51f6167752075/users/1cbc90af37634ccbbd1a1b7bf7b14fd5/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:11.205472 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f28f6e1-88de-4726-8db1-5bdbb4a03c7e admin admin] RBAC: Authorizing `identity:create_grant(project_id=a137d35b3eed42afb8f51f6167752075, user_id=1cbc90af37634ccbbd1a1b7bf7b14fd5, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:11.208133 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f28f6e1-88de-4726-8db1-5bdbb4a03c7e admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:11.219161 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1128/2250] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:08:11 2022] PUT /identity/v3/projects/a137d35b3eed42afb8f51f6167752075/users/1cbc90af37634ccbbd1a1b7bf7b14fd5/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:08:11.230296 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a97ddf02-f8b3-41e5-813f-c3fd44a49d1c None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:11.230614 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a97ddf02-f8b3-41e5-813f-c3fd44a49d1c None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:11.231499 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a97ddf02-f8b3-41e5-813f-c3fd44a49d1c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:11.259157 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-a97ddf02-f8b3-41e5-813f-c3fd44a49d1c None None] MFA Rules not processed for user `1cbc90af37634ccbbd1a1b7bf7b14fd5`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:08:11.260367 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-a97ddf02-f8b3-41e5-813f-c3fd44a49d1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:11.335896 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1123/2251] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:08:11 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:08:11.349701 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f28f6e1-88de-4726-8db1-5bdbb4a03c7e admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:11.351583 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-3f28f6e1-88de-4726-8db1-5bdbb4a03c7e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:11.415065 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f28f6e1-88de-4726-8db1-5bdbb4a03c7e admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:11.419830 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a50feae2-2bc2-4ef0-879a-2ddf631c7673 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:11.420629 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a50feae2-2bc2-4ef0-879a-2ddf631c7673 service cinder] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:11.420904 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a50feae2-2bc2-4ef0-879a-2ddf631c7673 service cinder] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:11.421173 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a50feae2-2bc2-4ef0-879a-2ddf631c7673 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:11.430693 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a50feae2-2bc2-4ef0-879a-2ddf631c7673 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:11.432159 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a50feae2-2bc2-4ef0-879a-2ddf631c7673 service cinder] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:11.443292 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1129/2252] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:08:11 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:08:11.833532 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a97ddf02-f8b3-41e5-813f-c3fd44a49d1c None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:11.891767 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:11.895887 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a97ddf02-f8b3-41e5-813f-c3fd44a49d1c None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:11.901135 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b0af616-6896-4ff7-a4ae-ae148e773c5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:11.901753 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b0af616-6896-4ff7-a4ae-ae148e773c5f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:11.902032 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b0af616-6896-4ff7-a4ae-ae148e773c5f admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:11.902306 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b0af616-6896-4ff7-a4ae-ae148e773c5f admin admin] PATH_INFO: `/v3/users/2a92f39668bd4de3ba40b34edac35759` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:11.903934 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b0af616-6896-4ff7-a4ae-ae148e773c5f admin admin] RBAC: Authorizing `identity:delete_user(user_id=2a92f39668bd4de3ba40b34edac35759)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:11.906137 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b0af616-6896-4ff7-a4ae-ae148e773c5f admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:11.951503 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-0b0af616-6896-4ff7-a4ae-ae148e773c5f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2a92f39668bd4de3ba40b34edac35759'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:11.958732 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-0b0af616-6896-4ff7-a4ae-ae148e773c5f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2a92f39668bd4de3ba40b34edac35759'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:11.971835 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-0b0af616-6896-4ff7-a4ae-ae148e773c5f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2a92f39668bd4de3ba40b34edac35759'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:11.979326 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-0b0af616-6896-4ff7-a4ae-ae148e773c5f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2a92f39668bd4de3ba40b34edac35759'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:11.980078 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-0b0af616-6896-4ff7-a4ae-ae148e773c5f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2a92f39668bd4de3ba40b34edac35759'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:11.986680 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1124/2253] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:08:11 2022] DELETE /identity/v3/users/2a92f39668bd4de3ba40b34edac35759 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:08:11.991217 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ac10dd-f53f-4165-8dd7-b7669dce243b None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:11.991500 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ac10dd-f53f-4165-8dd7-b7669dce243b None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:11.991772 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ac10dd-f53f-4165-8dd7-b7669dce243b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:12.010309 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-83ac10dd-f53f-4165-8dd7-b7669dce243b None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:08:12.011567 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-83ac10dd-f53f-4165-8dd7-b7669dce243b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:12.071423 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1130/2254] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:08:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 20 09:08:12.081719 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b0af616-6896-4ff7-a4ae-ae148e773c5f admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:12.083872 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-0b0af616-6896-4ff7-a4ae-ae148e773c5f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:12.141732 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b0af616-6896-4ff7-a4ae-ae148e773c5f admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:12.145396 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d05de1-4099-40db-83a1-34ccf6dfb2df service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:12.146002 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d05de1-4099-40db-83a1-34ccf6dfb2df service neutron] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:12.146272 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d05de1-4099-40db-83a1-34ccf6dfb2df service neutron] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:12.146569 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d05de1-4099-40db-83a1-34ccf6dfb2df service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:12.156421 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69d05de1-4099-40db-83a1-34ccf6dfb2df service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:12.157846 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69d05de1-4099-40db-83a1-34ccf6dfb2df service neutron] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:12.167307 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1125/2255] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:08:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 20 09:08:12.436074 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83ac10dd-f53f-4165-8dd7-b7669dce243b None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:12.437516 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-83ac10dd-f53f-4165-8dd7-b7669dce243b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:12.449222 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83ac10dd-f53f-4165-8dd7-b7669dce243b None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:12.454277 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b0188d9-888e-4311-a950-d2c2a274cca7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:12.454948 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b0188d9-888e-4311-a950-d2c2a274cca7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:12.455221 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b0188d9-888e-4311-a950-d2c2a274cca7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:12.455488 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b0188d9-888e-4311-a950-d2c2a274cca7 admin admin] PATH_INFO: `/v3/projects/d5af6df258994fc194d696afb5b384a8` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:12.456624 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b0188d9-888e-4311-a950-d2c2a274cca7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d5af6df258994fc194d696afb5b384a8)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:12.458551 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b0188d9-888e-4311-a950-d2c2a274cca7 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:12.506960 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-8b0188d9-888e-4311-a950-d2c2a274cca7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd5af6df258994fc194d696afb5b384a8', 'request_id': 'req-8b0188d9-888e-4311-a950-d2c2a274cca7'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:12.507977 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:INSERT=5,keystone:SELECT=91,keystone:DELETE=4 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:12.514874 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-8b0188d9-888e-4311-a950-d2c2a274cca7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd5af6df258994fc194d696afb5b384a8', 'request_id': 'req-8b0188d9-888e-4311-a950-d2c2a274cca7'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:12.521429 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-8b0188d9-888e-4311-a950-d2c2a274cca7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd5af6df258994fc194d696afb5b384a8', 'request_id': 'req-8b0188d9-888e-4311-a950-d2c2a274cca7'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:12.535321 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-8b0188d9-888e-4311-a950-d2c2a274cca7 admin admin] The token cache is being invalidate because project d5af6df258994fc194d696afb5b384a8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81561) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:08:12.535610 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-8b0188d9-888e-4311-a950-d2c2a274cca7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:12.536255 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-8b0188d9-888e-4311-a950-d2c2a274cca7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:12.537944 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1131/2256] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:08:12 2022] DELETE /identity/v3/projects/d5af6df258994fc194d696afb5b384a8 => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:08:12.541990 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d05de1-4099-40db-83a1-34ccf6dfb2df service neutron] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:12.552089 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d05de1-4099-40db-83a1-34ccf6dfb2df service neutron] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:12.607843 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c3b869-bc7d-44ef-94a9-d69af8b63938 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:12.608455 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c3b869-bc7d-44ef-94a9-d69af8b63938 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:12.608733 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c3b869-bc7d-44ef-94a9-d69af8b63938 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:12.609003 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c3b869-bc7d-44ef-94a9-d69af8b63938 admin admin] PATH_INFO: `/v3/users/1cbc90af37634ccbbd1a1b7bf7b14fd5` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:12.610106 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50c3b869-bc7d-44ef-94a9-d69af8b63938 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1cbc90af37634ccbbd1a1b7bf7b14fd5)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:12.612065 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50c3b869-bc7d-44ef-94a9-d69af8b63938 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:12.633811 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:12.660152 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-50c3b869-bc7d-44ef-94a9-d69af8b63938 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1cbc90af37634ccbbd1a1b7bf7b14fd5'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:12.667278 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-50c3b869-bc7d-44ef-94a9-d69af8b63938 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1cbc90af37634ccbbd1a1b7bf7b14fd5'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:12.679158 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-50c3b869-bc7d-44ef-94a9-d69af8b63938 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1cbc90af37634ccbbd1a1b7bf7b14fd5'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:12.686297 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-50c3b869-bc7d-44ef-94a9-d69af8b63938 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1cbc90af37634ccbbd1a1b7bf7b14fd5'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:12.686967 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-50c3b869-bc7d-44ef-94a9-d69af8b63938 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1cbc90af37634ccbbd1a1b7bf7b14fd5'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:12.692946 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1126/2257] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:08:12 2022] DELETE /identity/v3/users/1cbc90af37634ccbbd1a1b7bf7b14fd5 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:08:12.957909 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b0188d9-888e-4311-a950-d2c2a274cca7 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:12.959395 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-8b0188d9-888e-4311-a950-d2c2a274cca7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:12.974976 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b0188d9-888e-4311-a950-d2c2a274cca7 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:13.031807 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d84548f-de5c-469c-9f1b-5dd8a3c10b2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:13.032367 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d84548f-de5c-469c-9f1b-5dd8a3c10b2a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:13.032642 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d84548f-de5c-469c-9f1b-5dd8a3c10b2a admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:13.032939 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d84548f-de5c-469c-9f1b-5dd8a3c10b2a admin admin] PATH_INFO: `/v3/projects/a137d35b3eed42afb8f51f6167752075` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:13.034031 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d84548f-de5c-469c-9f1b-5dd8a3c10b2a admin admin] RBAC: Authorizing `identity:delete_project(project_id=a137d35b3eed42afb8f51f6167752075)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:13.035966 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d84548f-de5c-469c-9f1b-5dd8a3c10b2a admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:13.080866 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-5d84548f-de5c-469c-9f1b-5dd8a3c10b2a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a137d35b3eed42afb8f51f6167752075', 'request_id': 'req-5d84548f-de5c-469c-9f1b-5dd8a3c10b2a'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:13.086942 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-5d84548f-de5c-469c-9f1b-5dd8a3c10b2a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a137d35b3eed42afb8f51f6167752075', 'request_id': 'req-5d84548f-de5c-469c-9f1b-5dd8a3c10b2a'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:13.093270 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-5d84548f-de5c-469c-9f1b-5dd8a3c10b2a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a137d35b3eed42afb8f51f6167752075', 'request_id': 'req-5d84548f-de5c-469c-9f1b-5dd8a3c10b2a'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:13.105064 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-5d84548f-de5c-469c-9f1b-5dd8a3c10b2a admin admin] The token cache is being invalidate because project a137d35b3eed42afb8f51f6167752075 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81561) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:08:13.105348 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-5d84548f-de5c-469c-9f1b-5dd8a3c10b2a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:13.105986 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-5d84548f-de5c-469c-9f1b-5dd8a3c10b2a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:13.107496 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1132/2258] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:08:12 2022] DELETE /identity/v3/projects/a137d35b3eed42afb8f51f6167752075 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:08:13.112440 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a51611-f7a6-4905-8118-3ae21a0735cc None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:13.112898 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a51611-f7a6-4905-8118-3ae21a0735cc None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:13.113023 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a51611-f7a6-4905-8118-3ae21a0735cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:13.133824 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-c6a51611-f7a6-4905-8118-3ae21a0735cc None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:08:13.135128 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-c6a51611-f7a6-4905-8118-3ae21a0735cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:13.196232 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1127/2259] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:08:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:08:13.204056 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26526f4d-a3ed-48fe-b4c1-720f9bbe8590 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:13.204339 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26526f4d-a3ed-48fe-b4c1-720f9bbe8590 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:13.204606 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26526f4d-a3ed-48fe-b4c1-720f9bbe8590 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:13.223945 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-26526f4d-a3ed-48fe-b4c1-720f9bbe8590 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:08:13.225115 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-26526f4d-a3ed-48fe-b4c1-720f9bbe8590 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:13.233246 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1133/2260] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:08:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:08:13.237293 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a51611-f7a6-4905-8118-3ae21a0735cc None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:13.247382 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a51611-f7a6-4905-8118-3ae21a0735cc None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:13.251814 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bf410d2-6bf2-40ec-8cf5-634e73468d06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:13.252416 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf410d2-6bf2-40ec-8cf5-634e73468d06 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:13.252715 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf410d2-6bf2-40ec-8cf5-634e73468d06 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:13.252980 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf410d2-6bf2-40ec-8cf5-634e73468d06 admin admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:13.253612 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bf410d2-6bf2-40ec-8cf5-634e73468d06 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 09:08:13.253907 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bf410d2-6bf2-40ec-8cf5-634e73468d06 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:13.255524 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bf410d2-6bf2-40ec-8cf5-634e73468d06 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:13.266841 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1128/2261] 10.222.0.36 () {58 vars in 1153 bytes} [Thu Oct 20 09:08:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 09:08:13.271107 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26526f4d-a3ed-48fe-b4c1-720f9bbe8590 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:13.282135 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26526f4d-a3ed-48fe-b4c1-720f9bbe8590 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:13.287889 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-313d2175-268f-44ce-9c0c-a6593a951a21 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:13.288514 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313d2175-268f-44ce-9c0c-a6593a951a21 admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:13.288787 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313d2175-268f-44ce-9c0c-a6593a951a21 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:13.289054 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313d2175-268f-44ce-9c0c-a6593a951a21 admin admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:13.289939 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-313d2175-268f-44ce-9c0c-a6593a951a21 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:13.292176 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-313d2175-268f-44ce-9c0c-a6593a951a21 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:13.322048 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1134/2262] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:08:13 2022] POST /identity/v3/projects => generated 385 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:08:13.326321 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bf410d2-6bf2-40ec-8cf5-634e73468d06 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:13.336419 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bf410d2-6bf2-40ec-8cf5-634e73468d06 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:13.400227 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20031fb6-8d0c-461b-9386-db67eddca367 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:13.400835 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20031fb6-8d0c-461b-9386-db67eddca367 admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:13.401114 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20031fb6-8d0c-461b-9386-db67eddca367 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:13.401384 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20031fb6-8d0c-461b-9386-db67eddca367 admin admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:13.402080 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20031fb6-8d0c-461b-9386-db67eddca367 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:13.404568 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20031fb6-8d0c-461b-9386-db67eddca367 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:13.430944 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1129/2263] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:08:13 2022] POST /identity/v3/users => generated 293 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:08:13.434891 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-313d2175-268f-44ce-9c0c-a6593a951a21 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:13.445136 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-313d2175-268f-44ce-9c0c-a6593a951a21 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:13.449925 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e066fc21-79c6-4a2a-8984-6e1f9d87e249 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:13.450652 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e066fc21-79c6-4a2a-8984-6e1f9d87e249 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:13.450991 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e066fc21-79c6-4a2a-8984-6e1f9d87e249 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:13.451319 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e066fc21-79c6-4a2a-8984-6e1f9d87e249 admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:13.452017 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e066fc21-79c6-4a2a-8984-6e1f9d87e249 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:13.453707 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e066fc21-79c6-4a2a-8984-6e1f9d87e249 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:13.463904 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1135/2264] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:08:13 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 20 09:08:13.468656 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20031fb6-8d0c-461b-9386-db67eddca367 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:13.478742 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20031fb6-8d0c-461b-9386-db67eddca367 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:13.483947 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60baa22d-7810-42ff-b005-df752494cd9c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:13.484528 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60baa22d-7810-42ff-b005-df752494cd9c admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:13.484802 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60baa22d-7810-42ff-b005-df752494cd9c admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:13.485068 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60baa22d-7810-42ff-b005-df752494cd9c admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:13.485697 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60baa22d-7810-42ff-b005-df752494cd9c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:13.487791 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60baa22d-7810-42ff-b005-df752494cd9c admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:13.499060 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1130/2265] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:08:13 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:08:13.503176 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e066fc21-79c6-4a2a-8984-6e1f9d87e249 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:13.513683 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e066fc21-79c6-4a2a-8984-6e1f9d87e249 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:13.518193 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29663de9-a4d8-4b38-bc06-573aba02a543 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:13.518938 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29663de9-a4d8-4b38-bc06-573aba02a543 admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:13.519217 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29663de9-a4d8-4b38-bc06-573aba02a543 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:13.519485 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29663de9-a4d8-4b38-bc06-573aba02a543 admin admin] PATH_INFO: `/v3/projects/f046ebc25b8744b9b7c8b0727aa140d0/users/8541537127dc460b8e7a2d38094f7e6b/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:13.534547 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29663de9-a4d8-4b38-bc06-573aba02a543 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f046ebc25b8744b9b7c8b0727aa140d0, user_id=8541537127dc460b8e7a2d38094f7e6b, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:13.537136 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29663de9-a4d8-4b38-bc06-573aba02a543 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:13.550069 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1136/2266] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:08:13 2022] PUT /identity/v3/projects/f046ebc25b8744b9b7c8b0727aa140d0/users/8541537127dc460b8e7a2d38094f7e6b/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:08:13.560969 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f31ed7-af84-4773-a102-b4894288b003 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:13.561247 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f31ed7-af84-4773-a102-b4894288b003 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:13.561539 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f31ed7-af84-4773-a102-b4894288b003 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:13.569674 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:DELETE=7,keystone:SELECT=89,keystone:INSERT=4 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:13.592422 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-e0f31ed7-af84-4773-a102-b4894288b003 None None] MFA Rules not processed for user `8541537127dc460b8e7a2d38094f7e6b`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:08:13.593615 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-e0f31ed7-af84-4773-a102-b4894288b003 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:13.656798 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1131/2267] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:08:13 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 20 09:08:13.661438 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29663de9-a4d8-4b38-bc06-573aba02a543 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:13.667073 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:13.724708 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29663de9-a4d8-4b38-bc06-573aba02a543 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:13.729214 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eba26f00-5a5e-44a5-8fe2-abcee61522df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:13.729826 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba26f00-5a5e-44a5-8fe2-abcee61522df admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:13.730099 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba26f00-5a5e-44a5-8fe2-abcee61522df admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:13.730364 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba26f00-5a5e-44a5-8fe2-abcee61522df admin admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:13.731523 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eba26f00-5a5e-44a5-8fe2-abcee61522df admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:13.733413 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eba26f00-5a5e-44a5-8fe2-abcee61522df admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:13.759710 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1137/2268] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:08:13 2022] POST /identity/v3/projects => generated 387 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:08:13.763789 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0f31ed7-af84-4773-a102-b4894288b003 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:13.774394 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0f31ed7-af84-4773-a102-b4894288b003 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:13.830137 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d72495-f339-4c04-8837-e4f1de69e274 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:13.830700 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d72495-f339-4c04-8837-e4f1de69e274 admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:13.830982 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d72495-f339-4c04-8837-e4f1de69e274 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:13.831255 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d72495-f339-4c04-8837-e4f1de69e274 admin admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:13.831884 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0d72495-f339-4c04-8837-e4f1de69e274 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:13.833838 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0d72495-f339-4c04-8837-e4f1de69e274 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:13.855547 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1132/2269] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:08:13 2022] POST /identity/v3/users => generated 294 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:08:13.859391 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eba26f00-5a5e-44a5-8fe2-abcee61522df admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:13.869525 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eba26f00-5a5e-44a5-8fe2-abcee61522df admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:13.874375 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bd2f6ec-5430-4551-8c1e-1c28e95bfaa8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:13.875034 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd2f6ec-5430-4551-8c1e-1c28e95bfaa8 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:13.875324 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd2f6ec-5430-4551-8c1e-1c28e95bfaa8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:13.875612 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd2f6ec-5430-4551-8c1e-1c28e95bfaa8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:13.876279 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bd2f6ec-5430-4551-8c1e-1c28e95bfaa8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:13.877894 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bd2f6ec-5430-4551-8c1e-1c28e95bfaa8 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:13.889861 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1138/2270] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:08:13 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:08:13.893606 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d72495-f339-4c04-8837-e4f1de69e274 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:13.903979 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d72495-f339-4c04-8837-e4f1de69e274 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:13.908464 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45d6da10-4993-4718-8667-8ca869101709 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:13.909035 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d6da10-4993-4718-8667-8ca869101709 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:13.909308 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d6da10-4993-4718-8667-8ca869101709 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:13.909575 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d6da10-4993-4718-8667-8ca869101709 admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:13.910203 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45d6da10-4993-4718-8667-8ca869101709 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:13.911813 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45d6da10-4993-4718-8667-8ca869101709 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:13.921674 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1133/2271] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:08:13 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:08:13.925496 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bd2f6ec-5430-4551-8c1e-1c28e95bfaa8 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:13.936325 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bd2f6ec-5430-4551-8c1e-1c28e95bfaa8 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:13.940882 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-752e6427-fad8-405e-b821-787b197e8090 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:13.941519 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-752e6427-fad8-405e-b821-787b197e8090 admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:13.941798 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-752e6427-fad8-405e-b821-787b197e8090 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:13.942073 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-752e6427-fad8-405e-b821-787b197e8090 admin admin] PATH_INFO: `/v3/projects/c85f3a0d74d94c699fdd08d83f6db158/users/b37e182fb7ed46409049398832d13eaf/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:13.956941 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-752e6427-fad8-405e-b821-787b197e8090 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c85f3a0d74d94c699fdd08d83f6db158, user_id=b37e182fb7ed46409049398832d13eaf, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:13.959646 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-752e6427-fad8-405e-b821-787b197e8090 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:13.972990 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1139/2272] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:08:13 2022] PUT /identity/v3/projects/c85f3a0d74d94c699fdd08d83f6db158/users/b37e182fb7ed46409049398832d13eaf/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:08:13.984294 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4fe74db-5302-4746-81fa-e85a495deb29 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:13.984575 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4fe74db-5302-4746-81fa-e85a495deb29 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:13.984849 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4fe74db-5302-4746-81fa-e85a495deb29 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:14.012992 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-d4fe74db-5302-4746-81fa-e85a495deb29 None None] MFA Rules not processed for user `b37e182fb7ed46409049398832d13eaf`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:08:14.014184 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-d4fe74db-5302-4746-81fa-e85a495deb29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:14.076544 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1134/2273] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:08:13 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:08:14.081062 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-752e6427-fad8-405e-b821-787b197e8090 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:14.141524 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-752e6427-fad8-405e-b821-787b197e8090 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:14.146086 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d86eec87-d82e-486a-a708-f42aa54ee39e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:14.146699 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d86eec87-d82e-486a-a708-f42aa54ee39e admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:14.146979 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d86eec87-d82e-486a-a708-f42aa54ee39e admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:14.147251 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d86eec87-d82e-486a-a708-f42aa54ee39e admin admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:14.147878 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d86eec87-d82e-486a-a708-f42aa54ee39e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:14.149798 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d86eec87-d82e-486a-a708-f42aa54ee39e admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:14.176040 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1140/2274] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:08:14 2022] POST /identity/v3/projects => generated 385 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:08:14.180182 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4fe74db-5302-4746-81fa-e85a495deb29 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:14.186149 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:14.193925 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4fe74db-5302-4746-81fa-e85a495deb29 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:14.249515 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bd90e1f-f288-4748-a0ae-031708ccc484 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:14.250111 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bd90e1f-f288-4748-a0ae-031708ccc484 admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:14.250380 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bd90e1f-f288-4748-a0ae-031708ccc484 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:14.250673 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bd90e1f-f288-4748-a0ae-031708ccc484 admin admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:14.251293 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bd90e1f-f288-4748-a0ae-031708ccc484 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:14.253189 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bd90e1f-f288-4748-a0ae-031708ccc484 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:14.275606 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1135/2275] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:08:14 2022] POST /identity/v3/users => generated 299 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:08:14.279569 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d86eec87-d82e-486a-a708-f42aa54ee39e admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:14.289946 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d86eec87-d82e-486a-a708-f42aa54ee39e admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:14.294592 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f33c9c44-71fd-430c-9f3c-28f4f43c634f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:14.295181 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33c9c44-71fd-430c-9f3c-28f4f43c634f admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:14.295461 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33c9c44-71fd-430c-9f3c-28f4f43c634f admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:14.295760 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33c9c44-71fd-430c-9f3c-28f4f43c634f admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:14.296397 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f33c9c44-71fd-430c-9f3c-28f4f43c634f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:14.298006 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f33c9c44-71fd-430c-9f3c-28f4f43c634f admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:14.308441 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1141/2276] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:08:14 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:08:14.312942 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bd90e1f-f288-4748-a0ae-031708ccc484 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:14.322846 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bd90e1f-f288-4748-a0ae-031708ccc484 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:14.328294 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-835435fb-11fe-4865-a581-4694524b83b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:14.328959 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-835435fb-11fe-4865-a581-4694524b83b0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:14.329231 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-835435fb-11fe-4865-a581-4694524b83b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:14.329497 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-835435fb-11fe-4865-a581-4694524b83b0 admin admin] PATH_INFO: `/v3/projects/80d4e1a4753a486db4e241810148e7b5/users/a254c3b87bcf486f95f4b3c51d7a2846/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:14.345068 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-835435fb-11fe-4865-a581-4694524b83b0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=80d4e1a4753a486db4e241810148e7b5, user_id=a254c3b87bcf486f95f4b3c51d7a2846, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:14.347699 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-835435fb-11fe-4865-a581-4694524b83b0 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:14.360791 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1136/2277] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:08:14 2022] PUT /identity/v3/projects/80d4e1a4753a486db4e241810148e7b5/users/a254c3b87bcf486f95f4b3c51d7a2846/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:08:14.364877 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f33c9c44-71fd-430c-9f3c-28f4f43c634f admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:14.375029 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f33c9c44-71fd-430c-9f3c-28f4f43c634f admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:14.403567 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:14.433372 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8764cf82-88e9-4a43-8147-d6870524e750 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:14.434006 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8764cf82-88e9-4a43-8147-d6870524e750 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:14.434285 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8764cf82-88e9-4a43-8147-d6870524e750 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:14.434587 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8764cf82-88e9-4a43-8147-d6870524e750 admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:14.435220 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8764cf82-88e9-4a43-8147-d6870524e750 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:14.436821 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8764cf82-88e9-4a43-8147-d6870524e750 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:14.446382 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1142/2278] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:08:14 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:08:14.451706 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-835435fb-11fe-4865-a581-4694524b83b0 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:14.461458 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-835435fb-11fe-4865-a581-4694524b83b0 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:14.467528 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f03ef514-e422-424e-a73b-4de97060bfd1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:14.468178 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f03ef514-e422-424e-a73b-4de97060bfd1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:14.468454 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f03ef514-e422-424e-a73b-4de97060bfd1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:14.468728 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f03ef514-e422-424e-a73b-4de97060bfd1 admin admin] PATH_INFO: `/v3/projects/80d4e1a4753a486db4e241810148e7b5/users/a254c3b87bcf486f95f4b3c51d7a2846/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:14.470660 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f03ef514-e422-424e-a73b-4de97060bfd1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=80d4e1a4753a486db4e241810148e7b5, user_id=a254c3b87bcf486f95f4b3c51d7a2846, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:14.473464 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f03ef514-e422-424e-a73b-4de97060bfd1 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:14.485653 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1137/2279] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:08:14 2022] PUT /identity/v3/projects/80d4e1a4753a486db4e241810148e7b5/users/a254c3b87bcf486f95f4b3c51d7a2846/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:08:14.495819 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3039a363-4012-4ac1-b585-413606690e05 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:14.496102 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3039a363-4012-4ac1-b585-413606690e05 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:14.496961 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3039a363-4012-4ac1-b585-413606690e05 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:14.524692 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-3039a363-4012-4ac1-b585-413606690e05 None None] MFA Rules not processed for user `a254c3b87bcf486f95f4b3c51d7a2846`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:08:14.525902 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-3039a363-4012-4ac1-b585-413606690e05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:14.601094 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1143/2280] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:08:14 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:08:14.613197 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f03ef514-e422-424e-a73b-4de97060bfd1 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:14.615092 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-f03ef514-e422-424e-a73b-4de97060bfd1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:14.672035 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f03ef514-e422-424e-a73b-4de97060bfd1 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:14.675608 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d463646-ccc6-49da-ac9f-bb0492a4213c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:14.676194 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d463646-ccc6-49da-ac9f-bb0492a4213c service cinder] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:14.676463 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d463646-ccc6-49da-ac9f-bb0492a4213c service cinder] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:14.676725 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d463646-ccc6-49da-ac9f-bb0492a4213c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:14.686186 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d463646-ccc6-49da-ac9f-bb0492a4213c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:14.687632 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d463646-ccc6-49da-ac9f-bb0492a4213c service cinder] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:14.696950 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1138/2281] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:08:14 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:08:15.187985 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3039a363-4012-4ac1-b585-413606690e05 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:15.189594 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-3039a363-4012-4ac1-b585-413606690e05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:15.201516 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3039a363-4012-4ac1-b585-413606690e05 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:15.205133 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe7d345-8ecc-4790-9356-d9ac4ff5bebf service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:15.205732 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe7d345-8ecc-4790-9356-d9ac4ff5bebf service cinder] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:15.206012 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe7d345-8ecc-4790-9356-d9ac4ff5bebf service cinder] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:15.206286 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe7d345-8ecc-4790-9356-d9ac4ff5bebf service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:15.263392 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fe7d345-8ecc-4790-9356-d9ac4ff5bebf service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:15.264837 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fe7d345-8ecc-4790-9356-d9ac4ff5bebf service cinder] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:15.273744 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1144/2282] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:08:15 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 20 09:08:19.520672 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d463646-ccc6-49da-ac9f-bb0492a4213c service cinder] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:19.532780 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d463646-ccc6-49da-ac9f-bb0492a4213c service cinder] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:19.540017 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f069a22-1eca-42f3-b1e9-3badec80677a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:19.541129 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f069a22-1eca-42f3-b1e9-3badec80677a service cinder] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:19.541596 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f069a22-1eca-42f3-b1e9-3badec80677a service cinder] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:19.542150 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe7d345-8ecc-4790-9356-d9ac4ff5bebf service cinder] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:19.542655 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f069a22-1eca-42f3-b1e9-3badec80677a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:19.544384 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-9fe7d345-8ecc-4790-9356-d9ac4ff5bebf service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:19.612868 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:19.617292 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:19.618985 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f069a22-1eca-42f3-b1e9-3badec80677a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:19.621115 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f069a22-1eca-42f3-b1e9-3badec80677a service cinder] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:19.633175 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1139/2283] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:08:19 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:08:19.640334 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe7d345-8ecc-4790-9356-d9ac4ff5bebf service cinder] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:19.646433 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1ee264a-e696-4864-9369-1e540ab13ef1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:19.647313 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ee264a-e696-4864-9369-1e540ab13ef1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:19.647806 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ee264a-e696-4864-9369-1e540ab13ef1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:19.648282 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ee264a-e696-4864-9369-1e540ab13ef1 admin admin] PATH_INFO: `/v3/users/b52bc47bcce84c0c81aad4172106c04a` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:19.649495 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1ee264a-e696-4864-9369-1e540ab13ef1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b52bc47bcce84c0c81aad4172106c04a)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:19.651625 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1ee264a-e696-4864-9369-1e540ab13ef1 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:19.727519 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-c1ee264a-e696-4864-9369-1e540ab13ef1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b52bc47bcce84c0c81aad4172106c04a'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:19.758497 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-c1ee264a-e696-4864-9369-1e540ab13ef1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b52bc47bcce84c0c81aad4172106c04a'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:19.790581 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-c1ee264a-e696-4864-9369-1e540ab13ef1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b52bc47bcce84c0c81aad4172106c04a'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:19.800903 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-c1ee264a-e696-4864-9369-1e540ab13ef1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b52bc47bcce84c0c81aad4172106c04a'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:19.802219 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-c1ee264a-e696-4864-9369-1e540ab13ef1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b52bc47bcce84c0c81aad4172106c04a'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:19.813385 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1145/2284] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:08:19 2022] DELETE /identity/v3/users/b52bc47bcce84c0c81aad4172106c04a => generated 0 bytes in 269 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:08:19.816349 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6add129-7934-494b-9734-b23c6b376ba8 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:19.816657 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6add129-7934-494b-9734-b23c6b376ba8 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:19.817058 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6add129-7934-494b-9734-b23c6b376ba8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:19.844522 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-f6add129-7934-494b-9734-b23c6b376ba8 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:08:19.845751 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-f6add129-7934-494b-9734-b23c6b376ba8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:19.920339 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1140/2285] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:08:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:08:19.929260 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1ee264a-e696-4864-9369-1e540ab13ef1 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:19.930909 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-c1ee264a-e696-4864-9369-1e540ab13ef1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:19.996004 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1ee264a-e696-4864-9369-1e540ab13ef1 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:20.002829 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0100cd1f-dd5d-4c7c-810f-8e24b913bad3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:20.003535 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0100cd1f-dd5d-4c7c-810f-8e24b913bad3 service neutron] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:20.003813 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0100cd1f-dd5d-4c7c-810f-8e24b913bad3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:20.004079 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0100cd1f-dd5d-4c7c-810f-8e24b913bad3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:20.015085 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0100cd1f-dd5d-4c7c-810f-8e24b913bad3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:20.016522 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0100cd1f-dd5d-4c7c-810f-8e24b913bad3 service neutron] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:20.027173 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1146/2286] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:08:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:08:20.356332 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6add129-7934-494b-9734-b23c6b376ba8 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:20.357863 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-f6add129-7934-494b-9734-b23c6b376ba8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:20.385472 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6add129-7934-494b-9734-b23c6b376ba8 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:20.392760 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-078b6aed-425a-4c5e-b73c-6be7ba0afb24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:20.393445 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078b6aed-425a-4c5e-b73c-6be7ba0afb24 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:20.393725 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078b6aed-425a-4c5e-b73c-6be7ba0afb24 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:20.394873 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078b6aed-425a-4c5e-b73c-6be7ba0afb24 admin admin] PATH_INFO: `/v3/projects/52bb2568c8f04a1ba1d87a0d89c4bd05` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:20.395993 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-078b6aed-425a-4c5e-b73c-6be7ba0afb24 admin admin] RBAC: Authorizing `identity:delete_project(project_id=52bb2568c8f04a1ba1d87a0d89c4bd05)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:20.399653 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-078b6aed-425a-4c5e-b73c-6be7ba0afb24 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:20.450543 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-078b6aed-425a-4c5e-b73c-6be7ba0afb24 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '52bb2568c8f04a1ba1d87a0d89c4bd05', 'request_id': 'req-078b6aed-425a-4c5e-b73c-6be7ba0afb24'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:20.457073 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-078b6aed-425a-4c5e-b73c-6be7ba0afb24 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '52bb2568c8f04a1ba1d87a0d89c4bd05', 'request_id': 'req-078b6aed-425a-4c5e-b73c-6be7ba0afb24'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:20.462741 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-078b6aed-425a-4c5e-b73c-6be7ba0afb24 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '52bb2568c8f04a1ba1d87a0d89c4bd05', 'request_id': 'req-078b6aed-425a-4c5e-b73c-6be7ba0afb24'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:20.475789 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-078b6aed-425a-4c5e-b73c-6be7ba0afb24 admin admin] The token cache is being invalidate because project 52bb2568c8f04a1ba1d87a0d89c4bd05 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81562) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:08:20.476090 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-078b6aed-425a-4c5e-b73c-6be7ba0afb24 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:20.476695 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-078b6aed-425a-4c5e-b73c-6be7ba0afb24 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:20.478159 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1141/2287] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:08:20 2022] DELETE /identity/v3/projects/52bb2568c8f04a1ba1d87a0d89c4bd05 => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:08:20.482371 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0100cd1f-dd5d-4c7c-810f-8e24b913bad3 service neutron] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:20.492224 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0100cd1f-dd5d-4c7c-810f-8e24b913bad3 service neutron] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:20.549703 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42877f66-3baa-42df-988f-190113fce54b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:20.550445 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42877f66-3baa-42df-988f-190113fce54b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:20.550758 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42877f66-3baa-42df-988f-190113fce54b admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:20.551059 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42877f66-3baa-42df-988f-190113fce54b admin admin] PATH_INFO: `/v3/users/4869f672ff9e4a17a378b49a984f628c` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:20.552171 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42877f66-3baa-42df-988f-190113fce54b admin admin] RBAC: Authorizing `identity:delete_user(user_id=4869f672ff9e4a17a378b49a984f628c)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:20.554140 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42877f66-3baa-42df-988f-190113fce54b admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:20.569255 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:20.618988 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-42877f66-3baa-42df-988f-190113fce54b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4869f672ff9e4a17a378b49a984f628c'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:20.634586 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-42877f66-3baa-42df-988f-190113fce54b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4869f672ff9e4a17a378b49a984f628c'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:20.649108 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-42877f66-3baa-42df-988f-190113fce54b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4869f672ff9e4a17a378b49a984f628c'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:20.657131 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-42877f66-3baa-42df-988f-190113fce54b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4869f672ff9e4a17a378b49a984f628c'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:20.658614 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-42877f66-3baa-42df-988f-190113fce54b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4869f672ff9e4a17a378b49a984f628c'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:20.664781 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1147/2288] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:08:20 2022] DELETE /identity/v3/users/4869f672ff9e4a17a378b49a984f628c => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:08:20.985128 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-078b6aed-425a-4c5e-b73c-6be7ba0afb24 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:20.987407 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-078b6aed-425a-4c5e-b73c-6be7ba0afb24 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:21.012758 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-078b6aed-425a-4c5e-b73c-6be7ba0afb24 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:21.073309 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d55030fa-552e-4717-92e7-fe92fb3ef849 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:21.073946 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d55030fa-552e-4717-92e7-fe92fb3ef849 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:21.074786 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d55030fa-552e-4717-92e7-fe92fb3ef849 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:21.075077 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d55030fa-552e-4717-92e7-fe92fb3ef849 admin admin] PATH_INFO: `/v3/projects/61ee7d5324e84ec9990f5d2bd946f343` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:21.076201 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d55030fa-552e-4717-92e7-fe92fb3ef849 admin admin] RBAC: Authorizing `identity:delete_project(project_id=61ee7d5324e84ec9990f5d2bd946f343)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:21.078129 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d55030fa-552e-4717-92e7-fe92fb3ef849 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:21.101530 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:21.127568 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-d55030fa-552e-4717-92e7-fe92fb3ef849 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '61ee7d5324e84ec9990f5d2bd946f343', 'request_id': 'req-d55030fa-552e-4717-92e7-fe92fb3ef849'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:21.134078 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-d55030fa-552e-4717-92e7-fe92fb3ef849 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '61ee7d5324e84ec9990f5d2bd946f343', 'request_id': 'req-d55030fa-552e-4717-92e7-fe92fb3ef849'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:21.142077 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-d55030fa-552e-4717-92e7-fe92fb3ef849 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '61ee7d5324e84ec9990f5d2bd946f343', 'request_id': 'req-d55030fa-552e-4717-92e7-fe92fb3ef849'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:21.154712 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-d55030fa-552e-4717-92e7-fe92fb3ef849 admin admin] The token cache is being invalidate because project 61ee7d5324e84ec9990f5d2bd946f343 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81562) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:08:21.155040 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-d55030fa-552e-4717-92e7-fe92fb3ef849 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:21.155633 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-d55030fa-552e-4717-92e7-fe92fb3ef849 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:21.157213 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1142/2289] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:08:20 2022] DELETE /identity/v3/projects/61ee7d5324e84ec9990f5d2bd946f343 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:08:21.162593 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7566028f-6563-44da-8885-fe70af962493 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:21.162952 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7566028f-6563-44da-8885-fe70af962493 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:21.163414 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7566028f-6563-44da-8885-fe70af962493 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:21.185135 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-7566028f-6563-44da-8885-fe70af962493 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:08:21.185834 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-7566028f-6563-44da-8885-fe70af962493 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:21.254516 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1148/2290] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:08:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:08:21.263922 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc489c7-9bac-439f-914a-b6d6ab3efd77 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:21.264330 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc489c7-9bac-439f-914a-b6d6ab3efd77 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:21.264665 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc489c7-9bac-439f-914a-b6d6ab3efd77 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:21.290434 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-7fc489c7-9bac-439f-914a-b6d6ab3efd77 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:08:21.292074 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-7fc489c7-9bac-439f-914a-b6d6ab3efd77 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:21.302425 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1143/2291] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:08:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:08:21.307012 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7566028f-6563-44da-8885-fe70af962493 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:21.318121 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7566028f-6563-44da-8885-fe70af962493 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:21.323698 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1199649f-aa82-4130-b7b6-1a6b4ae049b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:21.324963 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1199649f-aa82-4130-b7b6-1a6b4ae049b8 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:21.325379 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1199649f-aa82-4130-b7b6-1a6b4ae049b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:21.325744 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1199649f-aa82-4130-b7b6-1a6b4ae049b8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:21.326456 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1199649f-aa82-4130-b7b6-1a6b4ae049b8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 09:08:21.326930 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1199649f-aa82-4130-b7b6-1a6b4ae049b8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:21.328634 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1199649f-aa82-4130-b7b6-1a6b4ae049b8 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:21.343361 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1149/2292] 10.222.0.36 () {58 vars in 1153 bytes} [Thu Oct 20 09:08:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 09:08:21.347209 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fc489c7-9bac-439f-914a-b6d6ab3efd77 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:21.357098 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fc489c7-9bac-439f-914a-b6d6ab3efd77 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:21.362140 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11211b24-8c90-4451-a6cc-97336c1bb139 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:21.362776 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11211b24-8c90-4451-a6cc-97336c1bb139 admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:21.363058 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11211b24-8c90-4451-a6cc-97336c1bb139 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:21.363348 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11211b24-8c90-4451-a6cc-97336c1bb139 admin admin] PATH_INFO: `/v3/projects` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:21.364017 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11211b24-8c90-4451-a6cc-97336c1bb139 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:21.365980 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11211b24-8c90-4451-a6cc-97336c1bb139 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:21.396000 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1144/2293] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:08:21 2022] POST /identity/v3/projects => generated 379 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 20 09:08:21.401379 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1199649f-aa82-4130-b7b6-1a6b4ae049b8 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:21.411965 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1199649f-aa82-4130-b7b6-1a6b4ae049b8 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:21.468596 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e4d8f70-fa31-4844-8057-aaf6ff135286 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:21.469196 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e4d8f70-fa31-4844-8057-aaf6ff135286 admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:21.469470 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e4d8f70-fa31-4844-8057-aaf6ff135286 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:21.469742 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e4d8f70-fa31-4844-8057-aaf6ff135286 admin admin] PATH_INFO: `/v3/users` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:21.470372 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e4d8f70-fa31-4844-8057-aaf6ff135286 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:21.472308 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e4d8f70-fa31-4844-8057-aaf6ff135286 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:21.494749 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1150/2294] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:08:21 2022] POST /identity/v3/users => generated 290 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:08:21.499480 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11211b24-8c90-4451-a6cc-97336c1bb139 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:21.509761 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11211b24-8c90-4451-a6cc-97336c1bb139 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:21.514428 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ad7c10-4e56-4959-9ef1-17bbd1169d87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:21.515125 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ad7c10-4e56-4959-9ef1-17bbd1169d87 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:21.515404 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ad7c10-4e56-4959-9ef1-17bbd1169d87 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:21.515672 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ad7c10-4e56-4959-9ef1-17bbd1169d87 admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:21.516335 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0ad7c10-4e56-4959-9ef1-17bbd1169d87 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:21.517929 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0ad7c10-4e56-4959-9ef1-17bbd1169d87 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:21.528319 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1145/2295] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:08:21 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:08:21.532263 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e4d8f70-fa31-4844-8057-aaf6ff135286 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:21.546095 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e4d8f70-fa31-4844-8057-aaf6ff135286 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:21.553193 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68009124-79d8-4bed-b595-ce9fafaeebd4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:21.553845 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68009124-79d8-4bed-b595-ce9fafaeebd4 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:21.554141 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68009124-79d8-4bed-b595-ce9fafaeebd4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:21.554491 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68009124-79d8-4bed-b595-ce9fafaeebd4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:21.555070 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68009124-79d8-4bed-b595-ce9fafaeebd4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:21.556668 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68009124-79d8-4bed-b595-ce9fafaeebd4 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:21.558100 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=87,keystone:DELETE=9,keystone:INSERT=4 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:21.570561 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1151/2296] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:08:21 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 20 09:08:21.574677 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ad7c10-4e56-4959-9ef1-17bbd1169d87 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:21.584826 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ad7c10-4e56-4959-9ef1-17bbd1169d87 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:21.589624 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c52b7ac-48b2-41d2-b3bc-c6cde8c6dac5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:21.590905 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c52b7ac-48b2-41d2-b3bc-c6cde8c6dac5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:21.591185 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c52b7ac-48b2-41d2-b3bc-c6cde8c6dac5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:21.591479 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c52b7ac-48b2-41d2-b3bc-c6cde8c6dac5 admin admin] PATH_INFO: `/v3/projects/a8375a12cea145f5a567bbb71344e928/users/acd654f725bd44a69a0080d87db1a9b6/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:21.607266 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c52b7ac-48b2-41d2-b3bc-c6cde8c6dac5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a8375a12cea145f5a567bbb71344e928, user_id=acd654f725bd44a69a0080d87db1a9b6, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:21.610056 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c52b7ac-48b2-41d2-b3bc-c6cde8c6dac5 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:21.622740 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1146/2297] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:08:21 2022] PUT /identity/v3/projects/a8375a12cea145f5a567bbb71344e928/users/acd654f725bd44a69a0080d87db1a9b6/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:08:21.635496 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1edffdfa-12ec-49c8-83ee-309ac0fcf060 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:21.635779 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1edffdfa-12ec-49c8-83ee-309ac0fcf060 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:21.636772 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1edffdfa-12ec-49c8-83ee-309ac0fcf060 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:21.667208 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-1edffdfa-12ec-49c8-83ee-309ac0fcf060 None None] MFA Rules not processed for user `acd654f725bd44a69a0080d87db1a9b6`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:08:21.668990 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-1edffdfa-12ec-49c8-83ee-309ac0fcf060 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:21.739404 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1152/2298] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:08:21 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:08:21.743979 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c52b7ac-48b2-41d2-b3bc-c6cde8c6dac5 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:21.838573 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c52b7ac-48b2-41d2-b3bc-c6cde8c6dac5 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:21.845991 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40129e92-f542-4b20-b3d4-6455f30c1a7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:21.846636 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40129e92-f542-4b20-b3d4-6455f30c1a7f admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:21.846924 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40129e92-f542-4b20-b3d4-6455f30c1a7f admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:21.847200 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40129e92-f542-4b20-b3d4-6455f30c1a7f admin admin] PATH_INFO: `/v3/projects` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:21.847859 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40129e92-f542-4b20-b3d4-6455f30c1a7f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:21.849813 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40129e92-f542-4b20-b3d4-6455f30c1a7f admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:21.879953 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1147/2299] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:08:21 2022] POST /identity/v3/projects => generated 379 bytes in 136 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:08:21.883820 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1edffdfa-12ec-49c8-83ee-309ac0fcf060 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:21.896219 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1edffdfa-12ec-49c8-83ee-309ac0fcf060 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:21.932309 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40129e92-f542-4b20-b3d4-6455f30c1a7f admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:21.934750 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-40129e92-f542-4b20-b3d4-6455f30c1a7f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:21.965218 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f259065a-cef1-4ba1-8ecd-e514acac5c9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:21.965883 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f259065a-cef1-4ba1-8ecd-e514acac5c9e admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:21.966245 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f259065a-cef1-4ba1-8ecd-e514acac5c9e admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:21.966494 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f259065a-cef1-4ba1-8ecd-e514acac5c9e admin admin] PATH_INFO: `/v3/users` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:21.967571 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f259065a-cef1-4ba1-8ecd-e514acac5c9e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:21.969165 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f259065a-cef1-4ba1-8ecd-e514acac5c9e admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:21.971157 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:DELETE=7,keystone:SELECT=89,keystone:INSERT=4 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:21.996971 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1153/2300] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:08:21 2022] POST /identity/v3/users => generated 290 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:08:22.000612 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f259065a-cef1-4ba1-8ecd-e514acac5c9e admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:22.010690 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f259065a-cef1-4ba1-8ecd-e514acac5c9e admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:22.015921 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-519cfe73-b99c-49d8-a388-5b160ed1cbbf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:22.016440 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40129e92-f542-4b20-b3d4-6455f30c1a7f admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:22.016910 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519cfe73-b99c-49d8-a388-5b160ed1cbbf admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:22.016910 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519cfe73-b99c-49d8-a388-5b160ed1cbbf admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:22.017066 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519cfe73-b99c-49d8-a388-5b160ed1cbbf admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:22.017716 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-519cfe73-b99c-49d8-a388-5b160ed1cbbf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:22.019504 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-519cfe73-b99c-49d8-a388-5b160ed1cbbf admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:22.021959 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f57ba71-595a-4cac-ab6c-0d574399b501 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:22.022918 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f57ba71-595a-4cac-ab6c-0d574399b501 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:22.023195 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f57ba71-595a-4cac-ab6c-0d574399b501 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:22.023463 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f57ba71-595a-4cac-ab6c-0d574399b501 admin admin] PATH_INFO: `/v3/users/8541537127dc460b8e7a2d38094f7e6b` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:22.024602 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f57ba71-595a-4cac-ab6c-0d574399b501 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8541537127dc460b8e7a2d38094f7e6b)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:22.027176 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f57ba71-595a-4cac-ab6c-0d574399b501 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:22.031176 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1154/2301] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:08:21 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:08:22.034840 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-519cfe73-b99c-49d8-a388-5b160ed1cbbf admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:22.045568 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-519cfe73-b99c-49d8-a388-5b160ed1cbbf admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:22.050733 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4db1696-c162-4681-ac13-768478a0b43e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:22.052007 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4db1696-c162-4681-ac13-768478a0b43e admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:22.052007 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4db1696-c162-4681-ac13-768478a0b43e admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:22.052007 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4db1696-c162-4681-ac13-768478a0b43e admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:22.052469 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4db1696-c162-4681-ac13-768478a0b43e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:22.054126 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4db1696-c162-4681-ac13-768478a0b43e admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:22.066851 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1155/2302] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:08:22 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 20 09:08:22.071707 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4db1696-c162-4681-ac13-768478a0b43e admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:22.083603 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4db1696-c162-4681-ac13-768478a0b43e admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:22.088892 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55826370-b084-45ba-9b6a-54299f42f141 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:22.089551 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55826370-b084-45ba-9b6a-54299f42f141 admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:22.090152 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55826370-b084-45ba-9b6a-54299f42f141 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:22.090398 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55826370-b084-45ba-9b6a-54299f42f141 admin admin] PATH_INFO: `/v3/projects/9523056e878c41769c7691d4120bf7fd/users/c2d59581bd2744db8baf160b299dc52a/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:22.092148 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-4f57ba71-595a-4cac-ab6c-0d574399b501 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8541537127dc460b8e7a2d38094f7e6b'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:22.101720 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-4f57ba71-595a-4cac-ab6c-0d574399b501 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8541537127dc460b8e7a2d38094f7e6b'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:22.107160 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55826370-b084-45ba-9b6a-54299f42f141 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9523056e878c41769c7691d4120bf7fd, user_id=c2d59581bd2744db8baf160b299dc52a, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:22.109841 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55826370-b084-45ba-9b6a-54299f42f141 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:22.117682 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-4f57ba71-595a-4cac-ab6c-0d574399b501 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8541537127dc460b8e7a2d38094f7e6b'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:22.124887 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1156/2303] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:08:22 2022] PUT /identity/v3/projects/9523056e878c41769c7691d4120bf7fd/users/c2d59581bd2744db8baf160b299dc52a/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:08:22.133866 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-4f57ba71-595a-4cac-ab6c-0d574399b501 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8541537127dc460b8e7a2d38094f7e6b'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:22.135609 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9192e16d-c596-4512-9aa5-171af3a590ab None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:22.136310 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9192e16d-c596-4512-9aa5-171af3a590ab None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:22.136523 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-4f57ba71-595a-4cac-ab6c-0d574399b501 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8541537127dc460b8e7a2d38094f7e6b'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:22.137331 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9192e16d-c596-4512-9aa5-171af3a590ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:22.144041 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1148/2304] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:08:21 2022] DELETE /identity/v3/users/8541537127dc460b8e7a2d38094f7e6b => generated 0 bytes in 212 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:08:22.148579 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3f2a95-2d77-4f4f-ac51-68a061fa13fa None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:22.148869 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3f2a95-2d77-4f4f-ac51-68a061fa13fa None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:22.149174 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3f2a95-2d77-4f4f-ac51-68a061fa13fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:22.169767 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-3e3f2a95-2d77-4f4f-ac51-68a061fa13fa None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:08:22.171172 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-3e3f2a95-2d77-4f4f-ac51-68a061fa13fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:22.171887 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-9192e16d-c596-4512-9aa5-171af3a590ab None None] MFA Rules not processed for user `c2d59581bd2744db8baf160b299dc52a`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:08:22.173044 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-9192e16d-c596-4512-9aa5-171af3a590ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:22.192872 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:22.236818 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1149/2305] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:08:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:08:22.237886 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e3f2a95-2d77-4f4f-ac51-68a061fa13fa None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:22.239278 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-3e3f2a95-2d77-4f4f-ac51-68a061fa13fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:22.244236 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1157/2306] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:08:22 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:08:22.249201 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9192e16d-c596-4512-9aa5-171af3a590ab None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:22.250732 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-9192e16d-c596-4512-9aa5-171af3a590ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:22.264341 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9192e16d-c596-4512-9aa5-171af3a590ab None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:22.269110 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2586c30e-f985-4d9d-a708-3df0cb388e4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:22.269709 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2586c30e-f985-4d9d-a708-3df0cb388e4c admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:22.269986 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2586c30e-f985-4d9d-a708-3df0cb388e4c admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:22.270260 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2586c30e-f985-4d9d-a708-3df0cb388e4c admin admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:22.270952 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2586c30e-f985-4d9d-a708-3df0cb388e4c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:22.272868 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2586c30e-f985-4d9d-a708-3df0cb388e4c admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:22.298655 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e3f2a95-2d77-4f4f-ac51-68a061fa13fa None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:22.305357 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1158/2307] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:08:22 2022] POST /identity/v3/projects => generated 379 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:08:22.305498 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c3bf11-6934-4a85-96db-4fa1362e9bfd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:22.305498 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c3bf11-6934-4a85-96db-4fa1362e9bfd service neutron] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:22.305706 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2586c30e-f985-4d9d-a708-3df0cb388e4c admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:22.305829 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c3bf11-6934-4a85-96db-4fa1362e9bfd service neutron] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:22.305829 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c3bf11-6934-4a85-96db-4fa1362e9bfd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:22.305829 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-46c3bf11-6934-4a85-96db-4fa1362e9bfd service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:22.307417 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-2586c30e-f985-4d9d-a708-3df0cb388e4c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:22.310614 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:22.364835 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2586c30e-f985-4d9d-a708-3df0cb388e4c admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:22.368402 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25146549-e283-4928-bc1e-503250eba90d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:22.369000 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25146549-e283-4928-bc1e-503250eba90d service neutron] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:22.369278 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25146549-e283-4928-bc1e-503250eba90d service neutron] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:22.369548 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25146549-e283-4928-bc1e-503250eba90d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:22.382577 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c3bf11-6934-4a85-96db-4fa1362e9bfd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:22.384043 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c3bf11-6934-4a85-96db-4fa1362e9bfd service neutron] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:22.394706 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1150/2308] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:08:22 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 157 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:08:22.395498 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c3bf11-6934-4a85-96db-4fa1362e9bfd service neutron] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:22.406453 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c3bf11-6934-4a85-96db-4fa1362e9bfd service neutron] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:22.454275 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25146549-e283-4928-bc1e-503250eba90d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:22.456427 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25146549-e283-4928-bc1e-503250eba90d service neutron] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:22.475297 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1159/2309] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:08:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 171 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:08:22.476639 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3292bd9-03ac-428f-83b0-07ba40f761c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:22.476639 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3292bd9-03ac-428f-83b0-07ba40f761c2 admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:22.476639 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3292bd9-03ac-428f-83b0-07ba40f761c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:22.476639 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3292bd9-03ac-428f-83b0-07ba40f761c2 admin admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:22.477182 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3292bd9-03ac-428f-83b0-07ba40f761c2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:22.477990 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3292bd9-03ac-428f-83b0-07ba40f761c2 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:22.511028 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1151/2310] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:08:22 2022] POST /identity/v3/users => generated 296 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:08:22.512627 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25146549-e283-4928-bc1e-503250eba90d service neutron] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:22.522906 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25146549-e283-4928-bc1e-503250eba90d service neutron] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:22.532127 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2626f9f3-8b50-4a65-9d5c-d12f2937273d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:22.532127 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2626f9f3-8b50-4a65-9d5c-d12f2937273d admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:22.532127 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2626f9f3-8b50-4a65-9d5c-d12f2937273d admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:22.532127 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2626f9f3-8b50-4a65-9d5c-d12f2937273d admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:22.534036 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2626f9f3-8b50-4a65-9d5c-d12f2937273d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:22.534036 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2626f9f3-8b50-4a65-9d5c-d12f2937273d admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:22.546671 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1160/2311] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:08:22 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:08:22.547687 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3292bd9-03ac-428f-83b0-07ba40f761c2 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:22.559314 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3292bd9-03ac-428f-83b0-07ba40f761c2 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:22.565674 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e079a5b5-07f8-469f-bd0e-f3a5111b9690 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:22.566336 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e079a5b5-07f8-469f-bd0e-f3a5111b9690 admin admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:22.566643 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e079a5b5-07f8-469f-bd0e-f3a5111b9690 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:22.567694 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e079a5b5-07f8-469f-bd0e-f3a5111b9690 admin admin] PATH_INFO: `/v3/projects/5a4bf4f26bb946ba98d4368af0541dbf/users/80e71045c27d4983a0137a2bcc18950a/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:22.587673 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e079a5b5-07f8-469f-bd0e-f3a5111b9690 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5a4bf4f26bb946ba98d4368af0541dbf, user_id=80e71045c27d4983a0137a2bcc18950a, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:22.591742 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e079a5b5-07f8-469f-bd0e-f3a5111b9690 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:22.609178 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1152/2312] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:08:22 2022] PUT /identity/v3/projects/5a4bf4f26bb946ba98d4368af0541dbf/users/80e71045c27d4983a0137a2bcc18950a/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 62 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:08:22.613469 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2626f9f3-8b50-4a65-9d5c-d12f2937273d admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:22.626862 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2626f9f3-8b50-4a65-9d5c-d12f2937273d admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:22.666637 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:22.695233 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-296618d3-a038-4557-9d25-38827b0bfa55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:22.695849 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-296618d3-a038-4557-9d25-38827b0bfa55 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:22.696126 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-296618d3-a038-4557-9d25-38827b0bfa55 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:22.696402 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-296618d3-a038-4557-9d25-38827b0bfa55 admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:22.697057 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-296618d3-a038-4557-9d25-38827b0bfa55 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:22.698813 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-296618d3-a038-4557-9d25-38827b0bfa55 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:22.709837 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1161/2313] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:08:22 2022] GET /identity/v3/roles => generated 1420 bytes in 97 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 20 09:08:22.714380 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e079a5b5-07f8-469f-bd0e-f3a5111b9690 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:22.726126 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e079a5b5-07f8-469f-bd0e-f3a5111b9690 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:22.732049 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19f19cde-2015-4c19-a751-70add9126c78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:22.732836 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19f19cde-2015-4c19-a751-70add9126c78 admin admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:22.733112 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19f19cde-2015-4c19-a751-70add9126c78 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:22.733409 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19f19cde-2015-4c19-a751-70add9126c78 admin admin] PATH_INFO: `/v3/projects/5a4bf4f26bb946ba98d4368af0541dbf/users/80e71045c27d4983a0137a2bcc18950a/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:22.735480 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19f19cde-2015-4c19-a751-70add9126c78 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5a4bf4f26bb946ba98d4368af0541dbf, user_id=80e71045c27d4983a0137a2bcc18950a, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:22.738442 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19f19cde-2015-4c19-a751-70add9126c78 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:22.749756 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1153/2314] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:08:22 2022] PUT /identity/v3/projects/5a4bf4f26bb946ba98d4368af0541dbf/users/80e71045c27d4983a0137a2bcc18950a/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:08:22.759847 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-113f4ca3-03e3-44ae-b11e-eef94b571ae4 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:22.760084 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-113f4ca3-03e3-44ae-b11e-eef94b571ae4 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:22.760353 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-113f4ca3-03e3-44ae-b11e-eef94b571ae4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:22.790319 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-113f4ca3-03e3-44ae-b11e-eef94b571ae4 None None] MFA Rules not processed for user `80e71045c27d4983a0137a2bcc18950a`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:08:22.791588 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-113f4ca3-03e3-44ae-b11e-eef94b571ae4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:22.880253 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1162/2315] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:08:22 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:08:22.895599 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19f19cde-2015-4c19-a751-70add9126c78 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:22.897535 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-19f19cde-2015-4c19-a751-70add9126c78 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:22.956087 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-113f4ca3-03e3-44ae-b11e-eef94b571ae4 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:22.957780 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-113f4ca3-03e3-44ae-b11e-eef94b571ae4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:22.981563 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19f19cde-2015-4c19-a751-70add9126c78 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:22.987673 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373ab91b-daed-4414-863a-822f4b888bab service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:22.988360 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373ab91b-daed-4414-863a-822f4b888bab service cinder] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:22.988566 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373ab91b-daed-4414-863a-822f4b888bab service cinder] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:22.989089 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373ab91b-daed-4414-863a-822f4b888bab service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:23.004472 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:23.007648 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-373ab91b-daed-4414-863a-822f4b888bab service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:23.009500 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-373ab91b-daed-4414-863a-822f4b888bab service cinder] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:23.024227 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1154/2316] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:08:22 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 129 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:08:23.046357 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-113f4ca3-03e3-44ae-b11e-eef94b571ae4 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:23.051871 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-310658ac-2b84-46e9-865b-e3c498b8fff4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:23.052624 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310658ac-2b84-46e9-865b-e3c498b8fff4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:23.053942 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310658ac-2b84-46e9-865b-e3c498b8fff4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:23.054234 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310658ac-2b84-46e9-865b-e3c498b8fff4 admin admin] PATH_INFO: `/v3/projects/f046ebc25b8744b9b7c8b0727aa140d0` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:23.055980 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-310658ac-2b84-46e9-865b-e3c498b8fff4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f046ebc25b8744b9b7c8b0727aa140d0)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:23.058806 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-310658ac-2b84-46e9-865b-e3c498b8fff4 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:23.079272 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373ab91b-daed-4414-863a-822f4b888bab service cinder] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:23.080770 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-373ab91b-daed-4414-863a-822f4b888bab service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:23.095052 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373ab91b-daed-4414-863a-822f4b888bab service cinder] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:23.100089 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-201a4e04-c01e-4a9e-aae4-a7e7c198d2b5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:23.100833 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-201a4e04-c01e-4a9e-aae4-a7e7c198d2b5 service cinder] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:23.101140 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-201a4e04-c01e-4a9e-aae4-a7e7c198d2b5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:23.101447 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-201a4e04-c01e-4a9e-aae4-a7e7c198d2b5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:23.103738 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-201a4e04-c01e-4a9e-aae4-a7e7c198d2b5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:23.125572 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-310658ac-2b84-46e9-865b-e3c498b8fff4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f046ebc25b8744b9b7c8b0727aa140d0', 'request_id': 'req-310658ac-2b84-46e9-865b-e3c498b8fff4'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:23.132004 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-310658ac-2b84-46e9-865b-e3c498b8fff4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f046ebc25b8744b9b7c8b0727aa140d0', 'request_id': 'req-310658ac-2b84-46e9-865b-e3c498b8fff4'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:23.136261 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:23.140438 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-310658ac-2b84-46e9-865b-e3c498b8fff4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f046ebc25b8744b9b7c8b0727aa140d0', 'request_id': 'req-310658ac-2b84-46e9-865b-e3c498b8fff4'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:23.156859 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-310658ac-2b84-46e9-865b-e3c498b8fff4 admin admin] The token cache is being invalidate because project f046ebc25b8744b9b7c8b0727aa140d0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81561) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:08:23.157203 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-310658ac-2b84-46e9-865b-e3c498b8fff4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:23.157792 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-310658ac-2b84-46e9-865b-e3c498b8fff4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:23.159651 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1163/2317] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:08:22 2022] DELETE /identity/v3/projects/f046ebc25b8744b9b7c8b0727aa140d0 => generated 0 bytes in 204 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:08:23.163784 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-310658ac-2b84-46e9-865b-e3c498b8fff4 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:23.166367 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-310658ac-2b84-46e9-865b-e3c498b8fff4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:23.180010 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-201a4e04-c01e-4a9e-aae4-a7e7c198d2b5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:23.181449 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-201a4e04-c01e-4a9e-aae4-a7e7c198d2b5 service cinder] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:23.196587 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1155/2318] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:08:23 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 20 09:08:23.244452 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-310658ac-2b84-46e9-865b-e3c498b8fff4 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:23.250098 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba0abc64-1d35-44d8-ae81-9e070079f018 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:23.250771 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba0abc64-1d35-44d8-ae81-9e070079f018 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:23.251036 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba0abc64-1d35-44d8-ae81-9e070079f018 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:23.251309 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba0abc64-1d35-44d8-ae81-9e070079f018 admin admin] PATH_INFO: `/v3/users/b37e182fb7ed46409049398832d13eaf` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:23.252986 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba0abc64-1d35-44d8-ae81-9e070079f018 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b37e182fb7ed46409049398832d13eaf)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:23.255575 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba0abc64-1d35-44d8-ae81-9e070079f018 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:23.314992 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-ba0abc64-1d35-44d8-ae81-9e070079f018 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b37e182fb7ed46409049398832d13eaf'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:23.325195 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-ba0abc64-1d35-44d8-ae81-9e070079f018 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b37e182fb7ed46409049398832d13eaf'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:23.339409 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-ba0abc64-1d35-44d8-ae81-9e070079f018 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b37e182fb7ed46409049398832d13eaf'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:23.351237 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-ba0abc64-1d35-44d8-ae81-9e070079f018 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b37e182fb7ed46409049398832d13eaf'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:23.351957 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-ba0abc64-1d35-44d8-ae81-9e070079f018 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b37e182fb7ed46409049398832d13eaf'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:23.359480 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1164/2319] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:08:23 2022] DELETE /identity/v3/users/b37e182fb7ed46409049398832d13eaf => generated 0 bytes in 196 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:08:23.762816 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-201a4e04-c01e-4a9e-aae4-a7e7c198d2b5 service cinder] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:23.801323 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba0abc64-1d35-44d8-ae81-9e070079f018 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:23.803059 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-ba0abc64-1d35-44d8-ae81-9e070079f018 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:23.829567 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-201a4e04-c01e-4a9e-aae4-a7e7c198d2b5 service cinder] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:23.891469 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba0abc64-1d35-44d8-ae81-9e070079f018 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:23.893929 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e09e98f6-637e-4990-995d-f7a2a58850bb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:23.894650 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e09e98f6-637e-4990-995d-f7a2a58850bb service neutron] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:23.894946 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e09e98f6-637e-4990-995d-f7a2a58850bb service neutron] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:23.895199 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e09e98f6-637e-4990-995d-f7a2a58850bb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:23.896951 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-e09e98f6-637e-4990-995d-f7a2a58850bb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:23.901423 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dfb3866-1309-491a-8d6c-23ec6fee4672 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:23.902880 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dfb3866-1309-491a-8d6c-23ec6fee4672 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:23.903349 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dfb3866-1309-491a-8d6c-23ec6fee4672 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:23.903778 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dfb3866-1309-491a-8d6c-23ec6fee4672 admin admin] PATH_INFO: `/v3/projects/c85f3a0d74d94c699fdd08d83f6db158` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:23.905639 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dfb3866-1309-491a-8d6c-23ec6fee4672 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c85f3a0d74d94c699fdd08d83f6db158)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:23.909250 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dfb3866-1309-491a-8d6c-23ec6fee4672 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:23.910186 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e09e98f6-637e-4990-995d-f7a2a58850bb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:23.911776 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e09e98f6-637e-4990-995d-f7a2a58850bb service neutron] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:23.921121 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:DELETE=12,keystone:SELECT=86,keystone:INSERT=2 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:23.923457 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1156/2320] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:08:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 161 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:08:23.979190 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-8dfb3866-1309-491a-8d6c-23ec6fee4672 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c85f3a0d74d94c699fdd08d83f6db158', 'request_id': 'req-8dfb3866-1309-491a-8d6c-23ec6fee4672'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:23.986817 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-8dfb3866-1309-491a-8d6c-23ec6fee4672 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c85f3a0d74d94c699fdd08d83f6db158', 'request_id': 'req-8dfb3866-1309-491a-8d6c-23ec6fee4672'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:23.994737 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-8dfb3866-1309-491a-8d6c-23ec6fee4672 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c85f3a0d74d94c699fdd08d83f6db158', 'request_id': 'req-8dfb3866-1309-491a-8d6c-23ec6fee4672'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:24.008783 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-8dfb3866-1309-491a-8d6c-23ec6fee4672 admin admin] The token cache is being invalidate because project c85f3a0d74d94c699fdd08d83f6db158 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81561) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:08:24.009118 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-8dfb3866-1309-491a-8d6c-23ec6fee4672 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:24.010066 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-8dfb3866-1309-491a-8d6c-23ec6fee4672 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:24.015461 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1165/2321] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:08:23 2022] DELETE /identity/v3/projects/c85f3a0d74d94c699fdd08d83f6db158 => generated 0 bytes in 212 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:08:24.016681 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e09e98f6-637e-4990-995d-f7a2a58850bb service neutron] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:24.026556 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e09e98f6-637e-4990-995d-f7a2a58850bb service neutron] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:24.084027 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:24.095939 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d82827b-549f-49cc-bcb1-0262d96790d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:24.096580 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d82827b-549f-49cc-bcb1-0262d96790d3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:24.096858 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d82827b-549f-49cc-bcb1-0262d96790d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:24.097127 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d82827b-549f-49cc-bcb1-0262d96790d3 admin admin] PATH_INFO: `/v3/users/a254c3b87bcf486f95f4b3c51d7a2846` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:24.098979 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d82827b-549f-49cc-bcb1-0262d96790d3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a254c3b87bcf486f95f4b3c51d7a2846)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:24.100326 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d82827b-549f-49cc-bcb1-0262d96790d3 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:24.154976 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-2d82827b-549f-49cc-bcb1-0262d96790d3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a254c3b87bcf486f95f4b3c51d7a2846'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:24.164805 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-2d82827b-549f-49cc-bcb1-0262d96790d3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a254c3b87bcf486f95f4b3c51d7a2846'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:24.182721 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-2d82827b-549f-49cc-bcb1-0262d96790d3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a254c3b87bcf486f95f4b3c51d7a2846'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:24.193249 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-2d82827b-549f-49cc-bcb1-0262d96790d3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a254c3b87bcf486f95f4b3c51d7a2846'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:24.194288 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-2d82827b-549f-49cc-bcb1-0262d96790d3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a254c3b87bcf486f95f4b3c51d7a2846'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:24.200588 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1157/2322] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:08:24 2022] DELETE /identity/v3/users/a254c3b87bcf486f95f4b3c51d7a2846 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:08:24.763262 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dfb3866-1309-491a-8d6c-23ec6fee4672 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:24.767464 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-8dfb3866-1309-491a-8d6c-23ec6fee4672 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:24.797231 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dfb3866-1309-491a-8d6c-23ec6fee4672 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:24.880956 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5906b09-594a-465f-9395-2529fcac7fab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:24.881612 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5906b09-594a-465f-9395-2529fcac7fab admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:24.882140 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5906b09-594a-465f-9395-2529fcac7fab admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:24.899830 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5906b09-594a-465f-9395-2529fcac7fab admin admin] PATH_INFO: `/v3/projects/80d4e1a4753a486db4e241810148e7b5` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:24.914607 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5906b09-594a-465f-9395-2529fcac7fab admin admin] RBAC: Authorizing `identity:delete_project(project_id=80d4e1a4753a486db4e241810148e7b5)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:24.917417 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5906b09-594a-465f-9395-2529fcac7fab admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:25.062890 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-a5906b09-594a-465f-9395-2529fcac7fab admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '80d4e1a4753a486db4e241810148e7b5', 'request_id': 'req-a5906b09-594a-465f-9395-2529fcac7fab'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:25.076222 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-a5906b09-594a-465f-9395-2529fcac7fab admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '80d4e1a4753a486db4e241810148e7b5', 'request_id': 'req-a5906b09-594a-465f-9395-2529fcac7fab'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:25.084048 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-a5906b09-594a-465f-9395-2529fcac7fab admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '80d4e1a4753a486db4e241810148e7b5', 'request_id': 'req-a5906b09-594a-465f-9395-2529fcac7fab'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:25.118355 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-a5906b09-594a-465f-9395-2529fcac7fab admin admin] The token cache is being invalidate because project 80d4e1a4753a486db4e241810148e7b5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81561) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:08:25.119108 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-a5906b09-594a-465f-9395-2529fcac7fab admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:25.120469 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-a5906b09-594a-465f-9395-2529fcac7fab admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:25.137790 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1166/2323] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:08:24 2022] DELETE /identity/v3/projects/80d4e1a4753a486db4e241810148e7b5 => generated 0 bytes in 374 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:08:25.142990 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72f0732b-237c-4770-98dd-12887c650b86 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:25.143484 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72f0732b-237c-4770-98dd-12887c650b86 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:25.143573 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72f0732b-237c-4770-98dd-12887c650b86 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:25.178823 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-72f0732b-237c-4770-98dd-12887c650b86 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:08:25.180157 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-72f0732b-237c-4770-98dd-12887c650b86 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:25.281656 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1158/2324] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:08:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:08:25.291969 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3239fa01-f45a-41f7-99ba-229a0c8e24f7 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:25.292362 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3239fa01-f45a-41f7-99ba-229a0c8e24f7 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:25.292790 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3239fa01-f45a-41f7-99ba-229a0c8e24f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:25.331534 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-3239fa01-f45a-41f7-99ba-229a0c8e24f7 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:08:25.332759 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-3239fa01-f45a-41f7-99ba-229a0c8e24f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:25.353980 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1167/2325] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:08:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 63 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:08:25.359474 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72f0732b-237c-4770-98dd-12887c650b86 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:25.370811 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72f0732b-237c-4770-98dd-12887c650b86 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:25.377525 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bc4154c-1ced-459a-9a80-2ceca4985b61 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:25.379318 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc4154c-1ced-459a-9a80-2ceca4985b61 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:25.379612 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc4154c-1ced-459a-9a80-2ceca4985b61 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:25.383098 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc4154c-1ced-459a-9a80-2ceca4985b61 admin admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:25.383412 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bc4154c-1ced-459a-9a80-2ceca4985b61 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 09:08:25.383723 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bc4154c-1ced-459a-9a80-2ceca4985b61 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:25.385746 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bc4154c-1ced-459a-9a80-2ceca4985b61 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:25.404591 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1159/2326] 10.222.0.36 () {58 vars in 1153 bytes} [Thu Oct 20 09:08:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 09:08:25.405612 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3239fa01-f45a-41f7-99ba-229a0c8e24f7 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:25.416140 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3239fa01-f45a-41f7-99ba-229a0c8e24f7 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:25.423301 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c403bb0-5f21-4f16-a1d2-0330a06c3d84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:25.424680 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c403bb0-5f21-4f16-a1d2-0330a06c3d84 admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:25.425486 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c403bb0-5f21-4f16-a1d2-0330a06c3d84 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:25.425486 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c403bb0-5f21-4f16-a1d2-0330a06c3d84 admin admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:25.426214 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c403bb0-5f21-4f16-a1d2-0330a06c3d84 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:25.429581 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c403bb0-5f21-4f16-a1d2-0330a06c3d84 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:25.462126 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:25.485658 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1168/2327] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:08:25 2022] POST /identity/v3/projects => generated 375 bytes in 78 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 20 09:08:25.486107 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bc4154c-1ced-459a-9a80-2ceca4985b61 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:25.499300 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bc4154c-1ced-459a-9a80-2ceca4985b61 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:25.575791 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de62894e-af0a-4006-9d36-5f95fd43c9b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:25.576440 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de62894e-af0a-4006-9d36-5f95fd43c9b4 admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:25.576718 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de62894e-af0a-4006-9d36-5f95fd43c9b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:25.577450 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de62894e-af0a-4006-9d36-5f95fd43c9b4 admin admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:25.578252 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de62894e-af0a-4006-9d36-5f95fd43c9b4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:25.580490 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de62894e-af0a-4006-9d36-5f95fd43c9b4 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:25.593563 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:25.616367 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1160/2328] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:08:25 2022] POST /identity/v3/users => generated 288 bytes in 128 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:08:25.617580 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c403bb0-5f21-4f16-a1d2-0330a06c3d84 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:25.629346 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c403bb0-5f21-4f16-a1d2-0330a06c3d84 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:25.634428 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b864c66-d50b-4af9-82be-5bfcb9cf9daf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:25.635255 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b864c66-d50b-4af9-82be-5bfcb9cf9daf admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:25.635634 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b864c66-d50b-4af9-82be-5bfcb9cf9daf admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:25.636004 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b864c66-d50b-4af9-82be-5bfcb9cf9daf admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:25.637495 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b864c66-d50b-4af9-82be-5bfcb9cf9daf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:25.639510 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b864c66-d50b-4af9-82be-5bfcb9cf9daf admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:25.656863 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1169/2329] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:08:25 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:08:25.658629 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de62894e-af0a-4006-9d36-5f95fd43c9b4 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:25.670167 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de62894e-af0a-4006-9d36-5f95fd43c9b4 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:25.684147 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1922459b-c6ce-4a04-a481-c5ea074ea832 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:25.684951 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1922459b-c6ce-4a04-a481-c5ea074ea832 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:25.685296 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1922459b-c6ce-4a04-a481-c5ea074ea832 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:25.685652 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1922459b-c6ce-4a04-a481-c5ea074ea832 admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:25.686405 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1922459b-c6ce-4a04-a481-c5ea074ea832 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:25.688109 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1922459b-c6ce-4a04-a481-c5ea074ea832 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:25.700783 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1161/2330] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:08:25 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:08:25.705566 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b864c66-d50b-4af9-82be-5bfcb9cf9daf admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:25.718366 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b864c66-d50b-4af9-82be-5bfcb9cf9daf admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:25.723789 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63b05853-30f9-4303-ab16-08e181a24d80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:25.725368 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b05853-30f9-4303-ab16-08e181a24d80 admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:25.725810 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b05853-30f9-4303-ab16-08e181a24d80 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:25.726251 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b05853-30f9-4303-ab16-08e181a24d80 admin admin] PATH_INFO: `/v3/projects/89787d6388634d8c91d894dfdeccd89d/users/50abe580d021427a954401e2615714a2/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:25.771192 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63b05853-30f9-4303-ab16-08e181a24d80 admin admin] RBAC: Authorizing `identity:create_grant(project_id=89787d6388634d8c91d894dfdeccd89d, user_id=50abe580d021427a954401e2615714a2, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:25.776145 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63b05853-30f9-4303-ab16-08e181a24d80 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:25.803789 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1170/2331] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:08:25 2022] PUT /identity/v3/projects/89787d6388634d8c91d894dfdeccd89d/users/50abe580d021427a954401e2615714a2/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 99 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:08:25.816385 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84700455-556c-4f28-ab12-3672a6453d5d None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:25.816732 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84700455-556c-4f28-ab12-3672a6453d5d None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:25.816929 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84700455-556c-4f28-ab12-3672a6453d5d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:25.849306 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-84700455-556c-4f28-ab12-3672a6453d5d None None] MFA Rules not processed for user `50abe580d021427a954401e2615714a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:08:25.850747 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-84700455-556c-4f28-ab12-3672a6453d5d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:25.948754 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1162/2332] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:08:25 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:08:25.962050 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63b05853-30f9-4303-ab16-08e181a24d80 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:25.964102 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-63b05853-30f9-4303-ab16-08e181a24d80 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:26.072743 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63b05853-30f9-4303-ab16-08e181a24d80 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:26.077174 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4294e56-45e4-48ef-ab08-56eef9bdfec3 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce043914cd9042909f70ab0adb272d34', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:26.078445 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4294e56-45e4-48ef-ab08-56eef9bdfec3 service glance] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:26.078445 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4294e56-45e4-48ef-ab08-56eef9bdfec3 service glance] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:26.078445 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4294e56-45e4-48ef-ab08-56eef9bdfec3 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:26.090956 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4294e56-45e4-48ef-ab08-56eef9bdfec3 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:26.092491 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4294e56-45e4-48ef-ab08-56eef9bdfec3 service glance] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:26.104789 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1171/2333] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:08:25 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 143 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:08:26.483690 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84700455-556c-4f28-ab12-3672a6453d5d None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:26.485232 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-84700455-556c-4f28-ab12-3672a6453d5d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:26.587832 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84700455-556c-4f28-ab12-3672a6453d5d None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:26.593828 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2444a607-5793-474d-9fd5-efafa0ef4402 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:26.594483 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2444a607-5793-474d-9fd5-efafa0ef4402 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:26.594767 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2444a607-5793-474d-9fd5-efafa0ef4402 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:26.595038 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2444a607-5793-474d-9fd5-efafa0ef4402 admin admin] PATH_INFO: `/v3/users/acd654f725bd44a69a0080d87db1a9b6` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:26.596170 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2444a607-5793-474d-9fd5-efafa0ef4402 admin admin] RBAC: Authorizing `identity:delete_user(user_id=acd654f725bd44a69a0080d87db1a9b6)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:26.598400 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2444a607-5793-474d-9fd5-efafa0ef4402 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:26.660323 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-2444a607-5793-474d-9fd5-efafa0ef4402 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'acd654f725bd44a69a0080d87db1a9b6'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:26.679122 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-2444a607-5793-474d-9fd5-efafa0ef4402 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'acd654f725bd44a69a0080d87db1a9b6'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:26.698312 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:INSERT=4,keystone:SELECT=86,keystone:DELETE=10 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:26.700715 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-2444a607-5793-474d-9fd5-efafa0ef4402 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'acd654f725bd44a69a0080d87db1a9b6'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:26.714129 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-2444a607-5793-474d-9fd5-efafa0ef4402 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'acd654f725bd44a69a0080d87db1a9b6'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:26.716645 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-2444a607-5793-474d-9fd5-efafa0ef4402 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'acd654f725bd44a69a0080d87db1a9b6'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:26.733291 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1163/2334] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:08:26 2022] DELETE /identity/v3/users/acd654f725bd44a69a0080d87db1a9b6 => generated 0 bytes in 250 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:08:26.737953 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34b11a3d-0797-4baa-a712-de6961dd0aab None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:26.738245 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34b11a3d-0797-4baa-a712-de6961dd0aab None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:26.738563 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34b11a3d-0797-4baa-a712-de6961dd0aab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:26.780827 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-34b11a3d-0797-4baa-a712-de6961dd0aab None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:08:26.782234 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-34b11a3d-0797-4baa-a712-de6961dd0aab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:26.878593 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1172/2335] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:08:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:08:26.888685 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2444a607-5793-474d-9fd5-efafa0ef4402 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:26.890069 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-2444a607-5793-474d-9fd5-efafa0ef4402 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:26.964478 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2444a607-5793-474d-9fd5-efafa0ef4402 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:26.968837 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a119b43-4db8-46c6-a2ee-fd2450e794e1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:26.969445 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a119b43-4db8-46c6-a2ee-fd2450e794e1 service neutron] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:26.969718 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a119b43-4db8-46c6-a2ee-fd2450e794e1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:26.969984 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a119b43-4db8-46c6-a2ee-fd2450e794e1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:26.979964 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a119b43-4db8-46c6-a2ee-fd2450e794e1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:26.981919 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a119b43-4db8-46c6-a2ee-fd2450e794e1 service neutron] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:26.995329 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1164/2336] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:08:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:08:27.343572 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34b11a3d-0797-4baa-a712-de6961dd0aab None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:27.345182 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-34b11a3d-0797-4baa-a712-de6961dd0aab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:27.359844 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34b11a3d-0797-4baa-a712-de6961dd0aab None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:27.370952 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa4208e-e4d4-4224-acfc-f32544d6c74f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:27.371600 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa4208e-e4d4-4224-acfc-f32544d6c74f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:27.372060 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa4208e-e4d4-4224-acfc-f32544d6c74f admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:27.372643 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa4208e-e4d4-4224-acfc-f32544d6c74f admin admin] PATH_INFO: `/v3/projects/a8375a12cea145f5a567bbb71344e928` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:27.381597 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fa4208e-e4d4-4224-acfc-f32544d6c74f admin admin] RBAC: Authorizing `identity:delete_project(project_id=a8375a12cea145f5a567bbb71344e928)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:27.383706 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fa4208e-e4d4-4224-acfc-f32544d6c74f admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:27.450111 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-5fa4208e-e4d4-4224-acfc-f32544d6c74f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a8375a12cea145f5a567bbb71344e928', 'request_id': 'req-5fa4208e-e4d4-4224-acfc-f32544d6c74f'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:27.457007 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-5fa4208e-e4d4-4224-acfc-f32544d6c74f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a8375a12cea145f5a567bbb71344e928', 'request_id': 'req-5fa4208e-e4d4-4224-acfc-f32544d6c74f'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:27.462374 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=93,keystone:INSERT=1,keystone:DELETE=6 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:27.463629 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-5fa4208e-e4d4-4224-acfc-f32544d6c74f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a8375a12cea145f5a567bbb71344e928', 'request_id': 'req-5fa4208e-e4d4-4224-acfc-f32544d6c74f'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:27.479050 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-5fa4208e-e4d4-4224-acfc-f32544d6c74f admin admin] The token cache is being invalidate because project a8375a12cea145f5a567bbb71344e928 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81561) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:08:27.479354 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-5fa4208e-e4d4-4224-acfc-f32544d6c74f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:27.480061 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-5fa4208e-e4d4-4224-acfc-f32544d6c74f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:27.481710 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1173/2337] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:08:27 2022] DELETE /identity/v3/projects/a8375a12cea145f5a567bbb71344e928 => generated 0 bytes in 139 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:08:27.485867 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a119b43-4db8-46c6-a2ee-fd2450e794e1 service neutron] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:27.498153 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a119b43-4db8-46c6-a2ee-fd2450e794e1 service neutron] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:27.570620 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef3f01b3-7831-4eb2-be09-8f3d92566f09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:27.571359 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef3f01b3-7831-4eb2-be09-8f3d92566f09 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:27.571705 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef3f01b3-7831-4eb2-be09-8f3d92566f09 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:27.572038 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef3f01b3-7831-4eb2-be09-8f3d92566f09 admin admin] PATH_INFO: `/v3/users/c2d59581bd2744db8baf160b299dc52a` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:27.573255 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef3f01b3-7831-4eb2-be09-8f3d92566f09 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c2d59581bd2744db8baf160b299dc52a)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:27.576010 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef3f01b3-7831-4eb2-be09-8f3d92566f09 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:27.650274 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-ef3f01b3-7831-4eb2-be09-8f3d92566f09 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c2d59581bd2744db8baf160b299dc52a'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:27.662441 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-ef3f01b3-7831-4eb2-be09-8f3d92566f09 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c2d59581bd2744db8baf160b299dc52a'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:27.686236 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-ef3f01b3-7831-4eb2-be09-8f3d92566f09 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c2d59581bd2744db8baf160b299dc52a'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:27.700569 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-ef3f01b3-7831-4eb2-be09-8f3d92566f09 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c2d59581bd2744db8baf160b299dc52a'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:27.701329 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-ef3f01b3-7831-4eb2-be09-8f3d92566f09 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c2d59581bd2744db8baf160b299dc52a'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:27.709428 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1165/2338] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:08:27 2022] DELETE /identity/v3/users/c2d59581bd2744db8baf160b299dc52a => generated 0 bytes in 224 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:08:28.256170 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa4208e-e4d4-4224-acfc-f32544d6c74f admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:28.259307 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-5fa4208e-e4d4-4224-acfc-f32544d6c74f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:28.261449 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef3f01b3-7831-4eb2-be09-8f3d92566f09 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:28.263274 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-ef3f01b3-7831-4eb2-be09-8f3d92566f09 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:28.275391 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa4208e-e4d4-4224-acfc-f32544d6c74f admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:28.275983 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:28.346761 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef3f01b3-7831-4eb2-be09-8f3d92566f09 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:28.348806 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-689ad2c5-17a3-47a2-94ec-a946c6da8393 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:28.349433 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-689ad2c5-17a3-47a2-94ec-a946c6da8393 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:28.349707 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-689ad2c5-17a3-47a2-94ec-a946c6da8393 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:28.349974 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-689ad2c5-17a3-47a2-94ec-a946c6da8393 admin admin] PATH_INFO: `/v3/users/fd9733d1bb2b4520b70090542615ddf9` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:28.351070 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-689ad2c5-17a3-47a2-94ec-a946c6da8393 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fd9733d1bb2b4520b70090542615ddf9)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:28.351282 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-474c5b93-1115-4185-871a-3edaf9d37522 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:28.351903 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-474c5b93-1115-4185-871a-3edaf9d37522 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:28.352181 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-474c5b93-1115-4185-871a-3edaf9d37522 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:28.352449 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-474c5b93-1115-4185-871a-3edaf9d37522 admin admin] PATH_INFO: `/v3/projects/9523056e878c41769c7691d4120bf7fd` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:28.352998 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-689ad2c5-17a3-47a2-94ec-a946c6da8393 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:28.353650 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-474c5b93-1115-4185-871a-3edaf9d37522 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9523056e878c41769c7691d4120bf7fd)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:28.356713 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-474c5b93-1115-4185-871a-3edaf9d37522 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:28.407496 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-689ad2c5-17a3-47a2-94ec-a946c6da8393 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fd9733d1bb2b4520b70090542615ddf9'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:28.411370 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-474c5b93-1115-4185-871a-3edaf9d37522 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9523056e878c41769c7691d4120bf7fd', 'request_id': 'req-474c5b93-1115-4185-871a-3edaf9d37522'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:28.415179 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-689ad2c5-17a3-47a2-94ec-a946c6da8393 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fd9733d1bb2b4520b70090542615ddf9'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:28.418238 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-474c5b93-1115-4185-871a-3edaf9d37522 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9523056e878c41769c7691d4120bf7fd', 'request_id': 'req-474c5b93-1115-4185-871a-3edaf9d37522'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:28.423993 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-474c5b93-1115-4185-871a-3edaf9d37522 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9523056e878c41769c7691d4120bf7fd', 'request_id': 'req-474c5b93-1115-4185-871a-3edaf9d37522'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:28.428413 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-689ad2c5-17a3-47a2-94ec-a946c6da8393 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fd9733d1bb2b4520b70090542615ddf9'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:28.436226 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-689ad2c5-17a3-47a2-94ec-a946c6da8393 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fd9733d1bb2b4520b70090542615ddf9'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:28.436884 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-689ad2c5-17a3-47a2-94ec-a946c6da8393 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fd9733d1bb2b4520b70090542615ddf9'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:28.438502 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-474c5b93-1115-4185-871a-3edaf9d37522 admin admin] The token cache is being invalidate because project 9523056e878c41769c7691d4120bf7fd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81562) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:08:28.438927 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-474c5b93-1115-4185-871a-3edaf9d37522 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:28.439649 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-474c5b93-1115-4185-871a-3edaf9d37522 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:28.441232 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1166/2339] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:08:28 2022] DELETE /identity/v3/projects/9523056e878c41769c7691d4120bf7fd => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:08:28.443146 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1174/2340] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:08:28 2022] DELETE /identity/v3/users/fd9733d1bb2b4520b70090542615ddf9 => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:08:28.448722 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-474c5b93-1115-4185-871a-3edaf9d37522 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:28.450726 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-474c5b93-1115-4185-871a-3edaf9d37522 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:28.521620 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-474c5b93-1115-4185-871a-3edaf9d37522 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:28.567453 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:28.586715 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa6cf115-db83-4bbf-9b0c-23149a0a7bb3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:28.587461 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa6cf115-db83-4bbf-9b0c-23149a0a7bb3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:28.587739 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa6cf115-db83-4bbf-9b0c-23149a0a7bb3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:28.588005 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa6cf115-db83-4bbf-9b0c-23149a0a7bb3 admin admin] PATH_INFO: `/v3/users/80e71045c27d4983a0137a2bcc18950a` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:28.589113 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa6cf115-db83-4bbf-9b0c-23149a0a7bb3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=80e71045c27d4983a0137a2bcc18950a)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:28.591884 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa6cf115-db83-4bbf-9b0c-23149a0a7bb3 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:28.629454 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-689ad2c5-17a3-47a2-94ec-a946c6da8393 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:28.631145 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-689ad2c5-17a3-47a2-94ec-a946c6da8393 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:28.648907 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-fa6cf115-db83-4bbf-9b0c-23149a0a7bb3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '80e71045c27d4983a0137a2bcc18950a'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:28.656503 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-689ad2c5-17a3-47a2-94ec-a946c6da8393 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:28.664050 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-fa6cf115-db83-4bbf-9b0c-23149a0a7bb3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '80e71045c27d4983a0137a2bcc18950a'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:28.665321 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2beba68d-74b6-489c-97ee-e2e78c8545f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:28.665975 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2beba68d-74b6-489c-97ee-e2e78c8545f0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:28.666273 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2beba68d-74b6-489c-97ee-e2e78c8545f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:28.666551 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2beba68d-74b6-489c-97ee-e2e78c8545f0 admin admin] PATH_INFO: `/v3/projects/869b5cf3a9264fd3b70ae656d7814cfb` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:28.668033 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2beba68d-74b6-489c-97ee-e2e78c8545f0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=869b5cf3a9264fd3b70ae656d7814cfb)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:28.670006 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2beba68d-74b6-489c-97ee-e2e78c8545f0 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:28.677560 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-fa6cf115-db83-4bbf-9b0c-23149a0a7bb3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '80e71045c27d4983a0137a2bcc18950a'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:28.684557 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-fa6cf115-db83-4bbf-9b0c-23149a0a7bb3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '80e71045c27d4983a0137a2bcc18950a'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:28.685341 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-fa6cf115-db83-4bbf-9b0c-23149a0a7bb3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '80e71045c27d4983a0137a2bcc18950a'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:28.692129 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1167/2341] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:08:28 2022] DELETE /identity/v3/users/80e71045c27d4983a0137a2bcc18950a => generated 0 bytes in 244 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:08:28.723988 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-2beba68d-74b6-489c-97ee-e2e78c8545f0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '869b5cf3a9264fd3b70ae656d7814cfb', 'request_id': 'req-2beba68d-74b6-489c-97ee-e2e78c8545f0'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:28.730425 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-2beba68d-74b6-489c-97ee-e2e78c8545f0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '869b5cf3a9264fd3b70ae656d7814cfb', 'request_id': 'req-2beba68d-74b6-489c-97ee-e2e78c8545f0'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:28.736448 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-2beba68d-74b6-489c-97ee-e2e78c8545f0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '869b5cf3a9264fd3b70ae656d7814cfb', 'request_id': 'req-2beba68d-74b6-489c-97ee-e2e78c8545f0'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:28.738112 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=81,keystone:INSERT=2,keystone:DELETE=17 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:28.754422 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-2beba68d-74b6-489c-97ee-e2e78c8545f0 admin admin] The token cache is being invalidate because project 869b5cf3a9264fd3b70ae656d7814cfb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81561) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:08:28.754762 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-2beba68d-74b6-489c-97ee-e2e78c8545f0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:28.755416 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-2beba68d-74b6-489c-97ee-e2e78c8545f0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:28.760441 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1175/2342] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:08:28 2022] DELETE /identity/v3/projects/869b5cf3a9264fd3b70ae656d7814cfb => generated 0 bytes in 128 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:08:28.762654 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59dfab2e-1e42-45d3-83c9-37399b45b3f0 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:28.762968 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59dfab2e-1e42-45d3-83c9-37399b45b3f0 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:28.763230 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59dfab2e-1e42-45d3-83c9-37399b45b3f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:28.784316 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-59dfab2e-1e42-45d3-83c9-37399b45b3f0 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:08:28.786122 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-59dfab2e-1e42-45d3-83c9-37399b45b3f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:28.854521 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1168/2343] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:08:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:08:28.859324 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f75b8d-e857-4b4d-92dd-779b97c885c4 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:28.859696 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f75b8d-e857-4b4d-92dd-779b97c885c4 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:28.860033 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f75b8d-e857-4b4d-92dd-779b97c885c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:28.880968 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-49f75b8d-e857-4b4d-92dd-779b97c885c4 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:08:28.882267 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-49f75b8d-e857-4b4d-92dd-779b97c885c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:28.891199 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1176/2344] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:08:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:08:28.895578 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59dfab2e-1e42-45d3-83c9-37399b45b3f0 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:28.907452 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59dfab2e-1e42-45d3-83c9-37399b45b3f0 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:28.913979 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-254855a9-083b-4d2d-bbe4-2b94c0b8988c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:28.918507 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254855a9-083b-4d2d-bbe4-2b94c0b8988c admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:28.918507 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254855a9-083b-4d2d-bbe4-2b94c0b8988c admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:28.918507 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254855a9-083b-4d2d-bbe4-2b94c0b8988c admin admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:28.918507 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-254855a9-083b-4d2d-bbe4-2b94c0b8988c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 09:08:28.918507 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-254855a9-083b-4d2d-bbe4-2b94c0b8988c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:28.919849 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-254855a9-083b-4d2d-bbe4-2b94c0b8988c admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:28.934264 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1169/2345] 10.222.0.36 () {58 vars in 1153 bytes} [Thu Oct 20 09:08:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 09:08:28.943898 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49f75b8d-e857-4b4d-92dd-779b97c885c4 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:28.956774 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49f75b8d-e857-4b4d-92dd-779b97c885c4 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:28.964855 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-024f8d6f-c9c3-446e-94fe-e06cb16401f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:28.966195 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-024f8d6f-c9c3-446e-94fe-e06cb16401f2 admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:28.966505 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-024f8d6f-c9c3-446e-94fe-e06cb16401f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:28.966846 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-024f8d6f-c9c3-446e-94fe-e06cb16401f2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:28.967516 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-024f8d6f-c9c3-446e-94fe-e06cb16401f2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:28.969913 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-024f8d6f-c9c3-446e-94fe-e06cb16401f2 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:28.992766 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-254855a9-083b-4d2d-bbe4-2b94c0b8988c admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:28.994839 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-254855a9-083b-4d2d-bbe4-2b94c0b8988c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:29.007451 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1177/2346] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:08:28 2022] POST /identity/v3/projects => generated 365 bytes in 64 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:08:29.011167 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-024f8d6f-c9c3-446e-94fe-e06cb16401f2 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:29.016632 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-254855a9-083b-4d2d-bbe4-2b94c0b8988c admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:29.074263 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:29.083780 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-024f8d6f-c9c3-446e-94fe-e06cb16401f2 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:29.086022 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9acf072-ea53-4f08-9a78-15e1255cb123 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:29.086794 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9acf072-ea53-4f08-9a78-15e1255cb123 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:29.087076 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9acf072-ea53-4f08-9a78-15e1255cb123 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:29.087350 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9acf072-ea53-4f08-9a78-15e1255cb123 admin admin] PATH_INFO: `/v3/projects/5a4bf4f26bb946ba98d4368af0541dbf` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:29.088229 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f28f8a9-a85c-467a-add8-e71cc2af70e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:29.088491 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9acf072-ea53-4f08-9a78-15e1255cb123 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5a4bf4f26bb946ba98d4368af0541dbf)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:29.088962 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f28f8a9-a85c-467a-add8-e71cc2af70e9 admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:29.089306 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f28f8a9-a85c-467a-add8-e71cc2af70e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:29.090436 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f28f8a9-a85c-467a-add8-e71cc2af70e9 admin admin] PATH_INFO: `/v3/users` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:29.091140 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f28f8a9-a85c-467a-add8-e71cc2af70e9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:29.091244 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9acf072-ea53-4f08-9a78-15e1255cb123 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:29.093169 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f28f8a9-a85c-467a-add8-e71cc2af70e9 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:29.116449 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1178/2347] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:08:29 2022] POST /identity/v3/users => generated 283 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 20 09:08:29.120391 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f28f8a9-a85c-467a-add8-e71cc2af70e9 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:29.131232 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f28f8a9-a85c-467a-add8-e71cc2af70e9 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:29.147023 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-f9acf072-ea53-4f08-9a78-15e1255cb123 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5a4bf4f26bb946ba98d4368af0541dbf', 'request_id': 'req-f9acf072-ea53-4f08-9a78-15e1255cb123'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:29.154688 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-f9acf072-ea53-4f08-9a78-15e1255cb123 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5a4bf4f26bb946ba98d4368af0541dbf', 'request_id': 'req-f9acf072-ea53-4f08-9a78-15e1255cb123'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:29.160866 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-f9acf072-ea53-4f08-9a78-15e1255cb123 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5a4bf4f26bb946ba98d4368af0541dbf', 'request_id': 'req-f9acf072-ea53-4f08-9a78-15e1255cb123'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:29.175235 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-f9acf072-ea53-4f08-9a78-15e1255cb123 admin admin] The token cache is being invalidate because project 5a4bf4f26bb946ba98d4368af0541dbf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81562) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:08:29.175546 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-f9acf072-ea53-4f08-9a78-15e1255cb123 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:29.176188 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-f9acf072-ea53-4f08-9a78-15e1255cb123 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:29.178354 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1170/2348] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:08:28 2022] DELETE /identity/v3/projects/5a4bf4f26bb946ba98d4368af0541dbf => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:08:29.185140 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822e5c4e-ac76-453f-a1e2-753c20761e4d None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:29.185542 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822e5c4e-ac76-453f-a1e2-753c20761e4d None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:29.185689 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822e5c4e-ac76-453f-a1e2-753c20761e4d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:29.200988 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cc88d29-916a-411d-b145-95233431b468 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:29.201617 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc88d29-916a-411d-b145-95233431b468 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:29.201887 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc88d29-916a-411d-b145-95233431b468 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:29.202158 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc88d29-916a-411d-b145-95233431b468 admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:29.202826 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cc88d29-916a-411d-b145-95233431b468 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:29.204517 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cc88d29-916a-411d-b145-95233431b468 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:29.211088 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-822e5c4e-ac76-453f-a1e2-753c20761e4d None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:08:29.212307 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-822e5c4e-ac76-453f-a1e2-753c20761e4d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:29.215847 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1179/2349] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:08:29 2022] GET /identity/v3/roles => generated 1420 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:08:29.223438 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cc88d29-916a-411d-b145-95233431b468 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:29.225341 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-4cc88d29-916a-411d-b145-95233431b468 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:29.234242 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1171/2350] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:08:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 51 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 20 09:08:29.242546 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa4b0f6-d32a-4a19-8b92-d315469ec1d9 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:29.242833 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa4b0f6-d32a-4a19-8b92-d315469ec1d9 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:29.243073 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cc88d29-916a-411d-b145-95233431b468 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:29.243184 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa4b0f6-d32a-4a19-8b92-d315469ec1d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:29.248472 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b9702a-7a7e-4cbc-8c42-5bb0cfeec250 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:29.249078 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b9702a-7a7e-4cbc-8c42-5bb0cfeec250 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:29.249355 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b9702a-7a7e-4cbc-8c42-5bb0cfeec250 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:29.249632 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b9702a-7a7e-4cbc-8c42-5bb0cfeec250 admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:29.250322 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06b9702a-7a7e-4cbc-8c42-5bb0cfeec250 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:29.251989 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06b9702a-7a7e-4cbc-8c42-5bb0cfeec250 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:29.263119 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1180/2351] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:08:29 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:08:29.263774 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-3aa4b0f6-d32a-4a19-8b92-d315469ec1d9 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:08:29.265012 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-3aa4b0f6-d32a-4a19-8b92-d315469ec1d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:29.267276 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b9702a-7a7e-4cbc-8c42-5bb0cfeec250 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:29.276693 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1172/2352] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:08:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 20 09:08:29.278550 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aa4b0f6-d32a-4a19-8b92-d315469ec1d9 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:29.281466 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b9702a-7a7e-4cbc-8c42-5bb0cfeec250 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:29.286447 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bff5aed-e8f1-4015-a6a9-28c929cd5a17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:29.287140 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bff5aed-e8f1-4015-a6a9-28c929cd5a17 admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:29.287414 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bff5aed-e8f1-4015-a6a9-28c929cd5a17 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:29.287770 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bff5aed-e8f1-4015-a6a9-28c929cd5a17 admin admin] PATH_INFO: `/v3/projects/6f8e62812b574b378b10d3d76ac9b43c/users/d698db91fcfd4a42a552f03b14b16e52/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:29.287901 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aa4b0f6-d32a-4a19-8b92-d315469ec1d9 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:29.292540 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad697121-74ca-4e8e-93c8-74e17f68b0bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:29.293295 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad697121-74ca-4e8e-93c8-74e17f68b0bf admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:29.293572 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad697121-74ca-4e8e-93c8-74e17f68b0bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:29.293837 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad697121-74ca-4e8e-93c8-74e17f68b0bf admin admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:29.294495 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad697121-74ca-4e8e-93c8-74e17f68b0bf admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 09:08:29.294912 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad697121-74ca-4e8e-93c8-74e17f68b0bf admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:29.296505 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad697121-74ca-4e8e-93c8-74e17f68b0bf admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:29.303610 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bff5aed-e8f1-4015-a6a9-28c929cd5a17 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6f8e62812b574b378b10d3d76ac9b43c, user_id=d698db91fcfd4a42a552f03b14b16e52, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:29.306789 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bff5aed-e8f1-4015-a6a9-28c929cd5a17 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:29.312036 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1173/2353] 10.222.0.36 () {58 vars in 1153 bytes} [Thu Oct 20 09:08:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 09:08:29.312962 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad697121-74ca-4e8e-93c8-74e17f68b0bf admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:29.325534 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad697121-74ca-4e8e-93c8-74e17f68b0bf admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:29.326864 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1181/2354] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:08:29 2022] PUT /identity/v3/projects/6f8e62812b574b378b10d3d76ac9b43c/users/d698db91fcfd4a42a552f03b14b16e52/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:08:29.338333 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec7214d-e473-4505-8d83-10220103787e None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:29.338642 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec7214d-e473-4505-8d83-10220103787e None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:29.338914 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec7214d-e473-4505-8d83-10220103787e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:29.365552 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:INSERT=6,keystone:DELETE=1,keystone:SELECT=93 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:29.368947 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-6ec7214d-e473-4505-8d83-10220103787e None None] MFA Rules not processed for user `d698db91fcfd4a42a552f03b14b16e52`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:08:29.376136 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-6ec7214d-e473-4505-8d83-10220103787e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:29.393859 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-999744d7-419b-4914-80f5-6446cf7ec84e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:29.394493 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999744d7-419b-4914-80f5-6446cf7ec84e admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:29.394788 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999744d7-419b-4914-80f5-6446cf7ec84e admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:29.395058 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999744d7-419b-4914-80f5-6446cf7ec84e admin admin] PATH_INFO: `/v3/projects` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:29.395691 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-999744d7-419b-4914-80f5-6446cf7ec84e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:29.397588 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-999744d7-419b-4914-80f5-6446cf7ec84e admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:29.431682 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1174/2355] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:08:29 2022] POST /identity/v3/projects => generated 373 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 20 09:08:29.435887 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-999744d7-419b-4914-80f5-6446cf7ec84e admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:29.444001 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1182/2356] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:08:29 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 20 09:08:29.455583 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ec7214d-e473-4505-8d83-10220103787e None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:29.457026 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-6ec7214d-e473-4505-8d83-10220103787e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:29.491372 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:29.512274 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-999744d7-419b-4914-80f5-6446cf7ec84e admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:29.517044 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37265818-2ec3-451b-854f-4c2b4df282f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:29.517617 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37265818-2ec3-451b-854f-4c2b4df282f7 admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:29.517892 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37265818-2ec3-451b-854f-4c2b4df282f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:29.518158 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37265818-2ec3-451b-854f-4c2b4df282f7 admin admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:29.518814 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37265818-2ec3-451b-854f-4c2b4df282f7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:29.519709 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ec7214d-e473-4505-8d83-10220103787e None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:29.520722 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37265818-2ec3-451b-854f-4c2b4df282f7 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:29.543064 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1175/2357] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:08:29 2022] POST /identity/v3/users => generated 287 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 20 09:08:29.547588 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37265818-2ec3-451b-854f-4c2b4df282f7 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:29.558125 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37265818-2ec3-451b-854f-4c2b4df282f7 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:29.564154 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0666fa5-e833-479e-8b8f-346a7747b9a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:29.564763 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0666fa5-e833-479e-8b8f-346a7747b9a9 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:29.565037 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0666fa5-e833-479e-8b8f-346a7747b9a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:29.565366 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0666fa5-e833-479e-8b8f-346a7747b9a9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:29.566033 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0666fa5-e833-479e-8b8f-346a7747b9a9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:29.567687 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0666fa5-e833-479e-8b8f-346a7747b9a9 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:29.573425 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c25addf8-e420-425f-9f35-c6192499c033 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:29.574620 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25addf8-e420-425f-9f35-c6192499c033 service cinder] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:29.574898 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25addf8-e420-425f-9f35-c6192499c033 service cinder] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:29.575163 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25addf8-e420-425f-9f35-c6192499c033 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:29.578534 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1176/2358] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:08:29 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:08:29.582583 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0666fa5-e833-479e-8b8f-346a7747b9a9 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:29.593071 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0666fa5-e833-479e-8b8f-346a7747b9a9 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:29.598122 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c603feb3-8105-40d8-8034-8136effff7cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:29.598777 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c603feb3-8105-40d8-8034-8136effff7cf admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:29.599052 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c603feb3-8105-40d8-8034-8136effff7cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:29.599320 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c603feb3-8105-40d8-8034-8136effff7cf admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:29.599973 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c603feb3-8105-40d8-8034-8136effff7cf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:29.601580 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c603feb3-8105-40d8-8034-8136effff7cf admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:29.613585 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1177/2359] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:08:29 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 20 09:08:29.617801 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c603feb3-8105-40d8-8034-8136effff7cf admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:29.627763 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c603feb3-8105-40d8-8034-8136effff7cf admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:29.633330 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a6c0ed5-7a7e-4aa4-8a85-cb6a613eeae2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:29.633942 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6c0ed5-7a7e-4aa4-8a85-cb6a613eeae2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:29.634222 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6c0ed5-7a7e-4aa4-8a85-cb6a613eeae2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:29.634641 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6c0ed5-7a7e-4aa4-8a85-cb6a613eeae2 admin admin] PATH_INFO: `/v3/projects/6bc433aee59d42cea6504cafd89750c8/users/d947ed034ad34c46933eb526c5883026/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:29.650351 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c25addf8-e420-425f-9f35-c6192499c033 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:29.651023 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a6c0ed5-7a7e-4aa4-8a85-cb6a613eeae2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6bc433aee59d42cea6504cafd89750c8, user_id=d947ed034ad34c46933eb526c5883026, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:29.651846 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c25addf8-e420-425f-9f35-c6192499c033 service cinder] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:29.654336 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a6c0ed5-7a7e-4aa4-8a85-cb6a613eeae2 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:29.661403 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1183/2360] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:08:29 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 206 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 20 09:08:29.667312 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1178/2361] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:08:29 2022] PUT /identity/v3/projects/6bc433aee59d42cea6504cafd89750c8/users/d947ed034ad34c46933eb526c5883026/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:08:29.678596 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88c6f33-454b-4501-a78b-631b9e9d9ea5 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:29.678934 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88c6f33-454b-4501-a78b-631b9e9d9ea5 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:29.679227 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88c6f33-454b-4501-a78b-631b9e9d9ea5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:29.705448 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:29.708196 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-b88c6f33-454b-4501-a78b-631b9e9d9ea5 None None] MFA Rules not processed for user `d947ed034ad34c46933eb526c5883026`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:08:29.710266 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-b88c6f33-454b-4501-a78b-631b9e9d9ea5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:29.781409 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1184/2362] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:08:29 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:08:29.792806 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a6c0ed5-7a7e-4aa4-8a85-cb6a613eeae2 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:29.794600 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-6a6c0ed5-7a7e-4aa4-8a85-cb6a613eeae2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:29.858513 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a6c0ed5-7a7e-4aa4-8a85-cb6a613eeae2 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:29.866769 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a142f9-9599-4849-8ec3-5ba597be6ab2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:29.867387 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a142f9-9599-4849-8ec3-5ba597be6ab2 service cinder] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:29.867657 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a142f9-9599-4849-8ec3-5ba597be6ab2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:29.867920 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a142f9-9599-4849-8ec3-5ba597be6ab2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:29.877708 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33a142f9-9599-4849-8ec3-5ba597be6ab2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:29.878809 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33a142f9-9599-4849-8ec3-5ba597be6ab2 service cinder] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:29.889778 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1179/2363] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:08:29 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:08:32.973616 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b88c6f33-454b-4501-a78b-631b9e9d9ea5 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:33.034988 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b88c6f33-454b-4501-a78b-631b9e9d9ea5 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:33.039380 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f09abb5f-2899-4c82-ba7b-66d7c012c945 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:33.039992 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f09abb5f-2899-4c82-ba7b-66d7c012c945 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:33.040266 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f09abb5f-2899-4c82-ba7b-66d7c012c945 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:33.040530 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f09abb5f-2899-4c82-ba7b-66d7c012c945 admin admin] PATH_INFO: `/v3/users/d947ed034ad34c46933eb526c5883026` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:33.041617 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f09abb5f-2899-4c82-ba7b-66d7c012c945 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d947ed034ad34c46933eb526c5883026)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:33.044178 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f09abb5f-2899-4c82-ba7b-66d7c012c945 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:33.091373 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-f09abb5f-2899-4c82-ba7b-66d7c012c945 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd947ed034ad34c46933eb526c5883026'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:33.099777 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-f09abb5f-2899-4c82-ba7b-66d7c012c945 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd947ed034ad34c46933eb526c5883026'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:33.112827 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-f09abb5f-2899-4c82-ba7b-66d7c012c945 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd947ed034ad34c46933eb526c5883026'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:33.119745 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-f09abb5f-2899-4c82-ba7b-66d7c012c945 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd947ed034ad34c46933eb526c5883026'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:33.120394 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-f09abb5f-2899-4c82-ba7b-66d7c012c945 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd947ed034ad34c46933eb526c5883026'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:33.126184 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1185/2364] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:08:32 2022] DELETE /identity/v3/users/d947ed034ad34c46933eb526c5883026 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:08:33.130869 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5ce75f3-a989-49e9-a62a-913c63e58892 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:33.131355 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5ce75f3-a989-49e9-a62a-913c63e58892 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:33.131480 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5ce75f3-a989-49e9-a62a-913c63e58892 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:33.193588 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-d5ce75f3-a989-49e9-a62a-913c63e58892 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:08:33.195228 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-d5ce75f3-a989-49e9-a62a-913c63e58892 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:33.246319 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:33.293658 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1180/2365] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:08:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:08:33.304306 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f09abb5f-2899-4c82-ba7b-66d7c012c945 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:33.305813 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-f09abb5f-2899-4c82-ba7b-66d7c012c945 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:33.332830 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:33.370173 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f09abb5f-2899-4c82-ba7b-66d7c012c945 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:33.374293 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bef53ba5-a0a1-495c-bad5-7be11d4b173f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:33.374931 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef53ba5-a0a1-495c-bad5-7be11d4b173f service neutron] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:33.375207 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef53ba5-a0a1-495c-bad5-7be11d4b173f service neutron] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:33.375474 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef53ba5-a0a1-495c-bad5-7be11d4b173f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:33.385228 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bef53ba5-a0a1-495c-bad5-7be11d4b173f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:33.386697 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bef53ba5-a0a1-495c-bad5-7be11d4b173f service neutron] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:33.397168 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1186/2366] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:08:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:08:33.669495 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5ce75f3-a989-49e9-a62a-913c63e58892 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:33.671285 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-d5ce75f3-a989-49e9-a62a-913c63e58892 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:33.686003 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5ce75f3-a989-49e9-a62a-913c63e58892 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:33.690809 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc373c0f-1378-4dc9-9185-b0075423d9c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:33.691509 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc373c0f-1378-4dc9-9185-b0075423d9c8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:33.691850 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc373c0f-1378-4dc9-9185-b0075423d9c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:33.692189 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc373c0f-1378-4dc9-9185-b0075423d9c8 admin admin] PATH_INFO: `/v3/projects/6bc433aee59d42cea6504cafd89750c8` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:33.693446 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc373c0f-1378-4dc9-9185-b0075423d9c8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6bc433aee59d42cea6504cafd89750c8)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:33.695315 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc373c0f-1378-4dc9-9185-b0075423d9c8 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:33.755008 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-cc373c0f-1378-4dc9-9185-b0075423d9c8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6bc433aee59d42cea6504cafd89750c8', 'request_id': 'req-cc373c0f-1378-4dc9-9185-b0075423d9c8'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:33.761227 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-cc373c0f-1378-4dc9-9185-b0075423d9c8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6bc433aee59d42cea6504cafd89750c8', 'request_id': 'req-cc373c0f-1378-4dc9-9185-b0075423d9c8'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:33.771360 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-cc373c0f-1378-4dc9-9185-b0075423d9c8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6bc433aee59d42cea6504cafd89750c8', 'request_id': 'req-cc373c0f-1378-4dc9-9185-b0075423d9c8'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:33.784434 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-cc373c0f-1378-4dc9-9185-b0075423d9c8 admin admin] The token cache is being invalidate because project 6bc433aee59d42cea6504cafd89750c8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81562) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:08:33.784719 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-cc373c0f-1378-4dc9-9185-b0075423d9c8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:33.785369 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-cc373c0f-1378-4dc9-9185-b0075423d9c8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:33.786906 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1181/2367] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:08:33 2022] DELETE /identity/v3/projects/6bc433aee59d42cea6504cafd89750c8 => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:08:33.791837 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad7f80df-65a7-467c-871f-286f82f8b663 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:33.792118 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad7f80df-65a7-467c-871f-286f82f8b663 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:33.792382 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad7f80df-65a7-467c-871f-286f82f8b663 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:33.846121 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-ad7f80df-65a7-467c-871f-286f82f8b663 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:08:33.847463 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-ad7f80df-65a7-467c-871f-286f82f8b663 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:33.911871 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1187/2368] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:08:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:08:33.921510 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c621678c-38d1-42d6-83ec-6a8851e9c7a0 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:33.921803 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c621678c-38d1-42d6-83ec-6a8851e9c7a0 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:33.922079 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c621678c-38d1-42d6-83ec-6a8851e9c7a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:33.943359 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-c621678c-38d1-42d6-83ec-6a8851e9c7a0 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:08:33.944583 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-c621678c-38d1-42d6-83ec-6a8851e9c7a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:33.953028 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1182/2369] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:08:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:08:33.957621 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad7f80df-65a7-467c-871f-286f82f8b663 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:33.970702 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad7f80df-65a7-467c-871f-286f82f8b663 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:33.977596 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82277162-7139-44c5-9b76-55a3a8780b20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:33.978210 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82277162-7139-44c5-9b76-55a3a8780b20 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:33.978516 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82277162-7139-44c5-9b76-55a3a8780b20 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:33.978795 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82277162-7139-44c5-9b76-55a3a8780b20 admin admin] PATH_INFO: `/v3/domains` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:33.979451 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82277162-7139-44c5-9b76-55a3a8780b20 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 09:08:33.979755 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82277162-7139-44c5-9b76-55a3a8780b20 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:33.981400 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82277162-7139-44c5-9b76-55a3a8780b20 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:33.993565 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1188/2370] 10.222.0.36 () {58 vars in 1153 bytes} [Thu Oct 20 09:08:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 09:08:33.998627 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c621678c-38d1-42d6-83ec-6a8851e9c7a0 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:34.008138 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c621678c-38d1-42d6-83ec-6a8851e9c7a0 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:34.013398 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a51a3781-4a4c-4788-ab86-fc988deff1d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:34.013992 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a51a3781-4a4c-4788-ab86-fc988deff1d1 admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:34.014263 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a51a3781-4a4c-4788-ab86-fc988deff1d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:34.014550 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a51a3781-4a4c-4788-ab86-fc988deff1d1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:34.015170 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a51a3781-4a4c-4788-ab86-fc988deff1d1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:34.017041 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a51a3781-4a4c-4788-ab86-fc988deff1d1 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:34.049870 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1183/2371] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:08:33 2022] POST /identity/v3/projects => generated 375 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:08:34.051070 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82277162-7139-44c5-9b76-55a3a8780b20 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:34.061556 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82277162-7139-44c5-9b76-55a3a8780b20 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:34.138727 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa2676c6-b75a-4f2b-9f6c-09bcecbb5db6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:34.139317 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa2676c6-b75a-4f2b-9f6c-09bcecbb5db6 admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:34.139591 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa2676c6-b75a-4f2b-9f6c-09bcecbb5db6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:34.139858 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa2676c6-b75a-4f2b-9f6c-09bcecbb5db6 admin admin] PATH_INFO: `/v3/users` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:34.140585 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa2676c6-b75a-4f2b-9f6c-09bcecbb5db6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:34.142372 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa2676c6-b75a-4f2b-9f6c-09bcecbb5db6 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:34.164808 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1189/2372] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:08:34 2022] POST /identity/v3/users => generated 288 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:08:34.169279 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a51a3781-4a4c-4788-ab86-fc988deff1d1 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:34.180489 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a51a3781-4a4c-4788-ab86-fc988deff1d1 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:34.185821 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12df31d6-0d97-449a-a8e8-fa3aee4fd869 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:34.186461 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12df31d6-0d97-449a-a8e8-fa3aee4fd869 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:34.186885 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12df31d6-0d97-449a-a8e8-fa3aee4fd869 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:34.187418 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12df31d6-0d97-449a-a8e8-fa3aee4fd869 admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:34.187883 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12df31d6-0d97-449a-a8e8-fa3aee4fd869 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:34.189508 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12df31d6-0d97-449a-a8e8-fa3aee4fd869 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:34.200553 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1184/2373] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:08:34 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:08:34.204702 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa2676c6-b75a-4f2b-9f6c-09bcecbb5db6 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:34.226985 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa2676c6-b75a-4f2b-9f6c-09bcecbb5db6 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:34.236769 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a29c3c0-dff9-4be1-b1b0-ee56fb905a55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:34.237468 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a29c3c0-dff9-4be1-b1b0-ee56fb905a55 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:34.237968 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a29c3c0-dff9-4be1-b1b0-ee56fb905a55 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:34.238245 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a29c3c0-dff9-4be1-b1b0-ee56fb905a55 admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:34.238927 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a29c3c0-dff9-4be1-b1b0-ee56fb905a55 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:34.240577 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a29c3c0-dff9-4be1-b1b0-ee56fb905a55 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:34.254253 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1190/2374] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:08:34 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:08:34.258290 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12df31d6-0d97-449a-a8e8-fa3aee4fd869 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:34.274846 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12df31d6-0d97-449a-a8e8-fa3aee4fd869 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:34.281896 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a48cd0eb-6c18-4709-9109-b1c3f710c6ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:34.282633 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a48cd0eb-6c18-4709-9109-b1c3f710c6ce admin admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:34.282975 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a48cd0eb-6c18-4709-9109-b1c3f710c6ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:34.283255 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a48cd0eb-6c18-4709-9109-b1c3f710c6ce admin admin] PATH_INFO: `/v3/projects/cdaedc6e2cd44c3a82ce641847fff8d9/users/851712bf63a04931bbc6925ebb3113ef/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:34.303604 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a48cd0eb-6c18-4709-9109-b1c3f710c6ce admin admin] RBAC: Authorizing `identity:create_grant(project_id=cdaedc6e2cd44c3a82ce641847fff8d9, user_id=851712bf63a04931bbc6925ebb3113ef, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:34.306570 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a48cd0eb-6c18-4709-9109-b1c3f710c6ce admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:34.322914 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1185/2375] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:08:34 2022] PUT /identity/v3/projects/cdaedc6e2cd44c3a82ce641847fff8d9/users/851712bf63a04931bbc6925ebb3113ef/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:08:34.333103 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22260f4a-d7de-4692-9aed-8d07e8a526ba None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:34.333391 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22260f4a-d7de-4692-9aed-8d07e8a526ba None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:34.333668 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22260f4a-d7de-4692-9aed-8d07e8a526ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:34.345609 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:34.374807 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-22260f4a-d7de-4692-9aed-8d07e8a526ba None None] MFA Rules not processed for user `851712bf63a04931bbc6925ebb3113ef`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:08:34.376251 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-22260f4a-d7de-4692-9aed-8d07e8a526ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:34.447269 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1191/2376] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:08:34 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 20 09:08:34.453111 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a48cd0eb-6c18-4709-9109-b1c3f710c6ce admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:34.499555 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:34.526964 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a48cd0eb-6c18-4709-9109-b1c3f710c6ce admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:34.532733 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad62ce13-2539-452f-a422-5b523c0ac126 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:34.533471 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad62ce13-2539-452f-a422-5b523c0ac126 admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:34.533787 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad62ce13-2539-452f-a422-5b523c0ac126 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:34.534113 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad62ce13-2539-452f-a422-5b523c0ac126 admin admin] PATH_INFO: `/v3/projects` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:34.534775 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad62ce13-2539-452f-a422-5b523c0ac126 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:34.536791 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad62ce13-2539-452f-a422-5b523c0ac126 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:34.567513 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1186/2377] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:08:34 2022] POST /identity/v3/projects => generated 375 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:08:34.571837 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22260f4a-d7de-4692-9aed-8d07e8a526ba None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:34.585462 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22260f4a-d7de-4692-9aed-8d07e8a526ba None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:34.644882 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd1f1909-dc20-42c3-98dc-9c54706cf221 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:34.645502 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd1f1909-dc20-42c3-98dc-9c54706cf221 admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:34.645779 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd1f1909-dc20-42c3-98dc-9c54706cf221 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:34.646049 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd1f1909-dc20-42c3-98dc-9c54706cf221 admin admin] PATH_INFO: `/v3/users` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:34.646706 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd1f1909-dc20-42c3-98dc-9c54706cf221 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:34.648798 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd1f1909-dc20-42c3-98dc-9c54706cf221 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:34.676177 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1192/2378] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:08:34 2022] POST /identity/v3/users => generated 288 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:08:34.677281 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad62ce13-2539-452f-a422-5b523c0ac126 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:34.688663 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad62ce13-2539-452f-a422-5b523c0ac126 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:34.693346 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1f793de-1de6-42c9-be16-406039142849 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:34.694064 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f793de-1de6-42c9-be16-406039142849 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:34.694213 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f793de-1de6-42c9-be16-406039142849 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:34.694520 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f793de-1de6-42c9-be16-406039142849 admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:34.695157 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1f793de-1de6-42c9-be16-406039142849 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:34.696856 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1f793de-1de6-42c9-be16-406039142849 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:34.708103 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1187/2379] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:08:34 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:08:34.712058 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd1f1909-dc20-42c3-98dc-9c54706cf221 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:34.724950 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd1f1909-dc20-42c3-98dc-9c54706cf221 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:34.729610 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-731b9bb4-ff3d-45c4-9c09-025073d6d71e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:34.730212 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731b9bb4-ff3d-45c4-9c09-025073d6d71e admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:34.730509 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731b9bb4-ff3d-45c4-9c09-025073d6d71e admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:34.730786 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731b9bb4-ff3d-45c4-9c09-025073d6d71e admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:34.732010 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-731b9bb4-ff3d-45c4-9c09-025073d6d71e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:34.733610 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-731b9bb4-ff3d-45c4-9c09-025073d6d71e admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:34.743733 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1193/2380] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:08:34 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:08:34.748782 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1f793de-1de6-42c9-be16-406039142849 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:34.762584 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1f793de-1de6-42c9-be16-406039142849 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:34.768300 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f05e8c8-f216-4a02-8031-86e622168780 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:34.768300 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f05e8c8-f216-4a02-8031-86e622168780 admin admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:34.768530 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f05e8c8-f216-4a02-8031-86e622168780 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:34.768711 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f05e8c8-f216-4a02-8031-86e622168780 admin admin] PATH_INFO: `/v3/projects/bb7baa8bfcbc49648399cac1bf51d820/users/487c94fdc1b94de7bda88116203924f7/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:34.785303 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f05e8c8-f216-4a02-8031-86e622168780 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bb7baa8bfcbc49648399cac1bf51d820, user_id=487c94fdc1b94de7bda88116203924f7, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:34.788001 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f05e8c8-f216-4a02-8031-86e622168780 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:34.803538 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1188/2381] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:08:34 2022] PUT /identity/v3/projects/bb7baa8bfcbc49648399cac1bf51d820/users/487c94fdc1b94de7bda88116203924f7/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:08:34.815429 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56efc13f-2b9f-4850-bab3-950709637a4b None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:34.815768 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56efc13f-2b9f-4850-bab3-950709637a4b None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:34.815971 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56efc13f-2b9f-4850-bab3-950709637a4b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:34.848979 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-56efc13f-2b9f-4850-bab3-950709637a4b None None] MFA Rules not processed for user `487c94fdc1b94de7bda88116203924f7`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:08:34.850271 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-56efc13f-2b9f-4850-bab3-950709637a4b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:34.916778 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:34.921025 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1194/2382] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:08:34 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:08:34.925713 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f05e8c8-f216-4a02-8031-86e622168780 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:34.986452 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f05e8c8-f216-4a02-8031-86e622168780 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:34.991006 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a82ae5e-7e7c-4da4-8898-76c184e5766c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:34.991590 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a82ae5e-7e7c-4da4-8898-76c184e5766c admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:34.991865 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a82ae5e-7e7c-4da4-8898-76c184e5766c admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:34.992132 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a82ae5e-7e7c-4da4-8898-76c184e5766c admin admin] PATH_INFO: `/v3/projects` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:34.992739 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a82ae5e-7e7c-4da4-8898-76c184e5766c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:34.994658 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a82ae5e-7e7c-4da4-8898-76c184e5766c admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:35.024354 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1189/2383] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:08:34 2022] POST /identity/v3/projects => generated 375 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:08:35.025637 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56efc13f-2b9f-4850-bab3-950709637a4b None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:35.036477 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56efc13f-2b9f-4850-bab3-950709637a4b None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:35.093778 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e983a9c-ae4d-4559-81b4-32881c03d601 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:35.094388 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e983a9c-ae4d-4559-81b4-32881c03d601 admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:35.094686 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e983a9c-ae4d-4559-81b4-32881c03d601 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:35.094990 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e983a9c-ae4d-4559-81b4-32881c03d601 admin admin] PATH_INFO: `/v3/users` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:35.095894 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e983a9c-ae4d-4559-81b4-32881c03d601 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:35.098202 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e983a9c-ae4d-4559-81b4-32881c03d601 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:35.119648 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1195/2384] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:08:35 2022] POST /identity/v3/users => generated 294 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:08:35.124821 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a82ae5e-7e7c-4da4-8898-76c184e5766c admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:35.136219 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a82ae5e-7e7c-4da4-8898-76c184e5766c admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:35.141744 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee468f98-96c3-4820-9cc9-2a13cb8ed8b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:35.142358 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee468f98-96c3-4820-9cc9-2a13cb8ed8b3 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:35.142669 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee468f98-96c3-4820-9cc9-2a13cb8ed8b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:35.142967 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee468f98-96c3-4820-9cc9-2a13cb8ed8b3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:35.143606 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee468f98-96c3-4820-9cc9-2a13cb8ed8b3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:35.145207 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee468f98-96c3-4820-9cc9-2a13cb8ed8b3 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:35.156911 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1190/2385] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:08:35 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:08:35.161507 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e983a9c-ae4d-4559-81b4-32881c03d601 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:35.172377 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e983a9c-ae4d-4559-81b4-32881c03d601 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:35.177105 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1817ed19-eaa8-43b1-a4d3-561e9092f044 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:35.177778 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1817ed19-eaa8-43b1-a4d3-561e9092f044 admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:35.178055 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1817ed19-eaa8-43b1-a4d3-561e9092f044 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:35.178323 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1817ed19-eaa8-43b1-a4d3-561e9092f044 admin admin] PATH_INFO: `/v3/projects/3a67e0084bba4195b8e721ffc451b793/users/5dd5dab6e7014daf8db03a631958ebd1/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:35.193480 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1817ed19-eaa8-43b1-a4d3-561e9092f044 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3a67e0084bba4195b8e721ffc451b793, user_id=5dd5dab6e7014daf8db03a631958ebd1, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:35.196222 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1817ed19-eaa8-43b1-a4d3-561e9092f044 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:35.210630 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1196/2386] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:08:35 2022] PUT /identity/v3/projects/3a67e0084bba4195b8e721ffc451b793/users/5dd5dab6e7014daf8db03a631958ebd1/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:08:35.215908 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee468f98-96c3-4820-9cc9-2a13cb8ed8b3 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:35.229021 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee468f98-96c3-4820-9cc9-2a13cb8ed8b3 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:35.289181 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa635260-5fc5-4fb4-b927-549f8f5d707a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:35.289815 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa635260-5fc5-4fb4-b927-549f8f5d707a admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:35.290224 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa635260-5fc5-4fb4-b927-549f8f5d707a admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:35.290389 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa635260-5fc5-4fb4-b927-549f8f5d707a admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:35.291024 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa635260-5fc5-4fb4-b927-549f8f5d707a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:35.292616 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa635260-5fc5-4fb4-b927-549f8f5d707a admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:35.298736 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:35.312461 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1191/2387] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:08:35 2022] GET /identity/v3/roles => generated 1420 bytes in 98 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:08:35.314034 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1817ed19-eaa8-43b1-a4d3-561e9092f044 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:35.325424 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1817ed19-eaa8-43b1-a4d3-561e9092f044 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:35.330298 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb49ce3-1712-48f7-afc0-34f76baa1d49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:35.331048 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb49ce3-1712-48f7-afc0-34f76baa1d49 admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:35.331331 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb49ce3-1712-48f7-afc0-34f76baa1d49 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:35.331596 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb49ce3-1712-48f7-afc0-34f76baa1d49 admin admin] PATH_INFO: `/v3/projects/3a67e0084bba4195b8e721ffc451b793/users/5dd5dab6e7014daf8db03a631958ebd1/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:35.333509 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebb49ce3-1712-48f7-afc0-34f76baa1d49 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3a67e0084bba4195b8e721ffc451b793, user_id=5dd5dab6e7014daf8db03a631958ebd1, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:35.336227 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebb49ce3-1712-48f7-afc0-34f76baa1d49 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:35.347576 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1197/2388] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:08:35 2022] PUT /identity/v3/projects/3a67e0084bba4195b8e721ffc451b793/users/5dd5dab6e7014daf8db03a631958ebd1/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:08:35.415974 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e19b13-5c3e-42bb-9e38-1a388da21f9b None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:35.416257 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e19b13-5c3e-42bb-9e38-1a388da21f9b None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:35.416531 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e19b13-5c3e-42bb-9e38-1a388da21f9b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:35.445869 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-b3e19b13-5c3e-42bb-9e38-1a388da21f9b None None] MFA Rules not processed for user `5dd5dab6e7014daf8db03a631958ebd1`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:08:35.447101 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-b3e19b13-5c3e-42bb-9e38-1a388da21f9b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:35.521011 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1192/2389] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:08:35 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:08:35.532963 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb49ce3-1712-48f7-afc0-34f76baa1d49 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:35.534721 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-ebb49ce3-1712-48f7-afc0-34f76baa1d49 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:35.611561 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb49ce3-1712-48f7-afc0-34f76baa1d49 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:35.615365 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0119c96e-f48d-4a90-a9d8-1b727973d389 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:35.615968 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0119c96e-f48d-4a90-a9d8-1b727973d389 service cinder] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:35.616244 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0119c96e-f48d-4a90-a9d8-1b727973d389 service cinder] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:35.616510 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0119c96e-f48d-4a90-a9d8-1b727973d389 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:35.665575 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:35.676526 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0119c96e-f48d-4a90-a9d8-1b727973d389 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:35.678042 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0119c96e-f48d-4a90-a9d8-1b727973d389 service cinder] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:35.688232 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1198/2390] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:08:35 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 156 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:08:37.233382 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e19b13-5c3e-42bb-9e38-1a388da21f9b None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:37.234916 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-b3e19b13-5c3e-42bb-9e38-1a388da21f9b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:37.249279 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e19b13-5c3e-42bb-9e38-1a388da21f9b None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:37.253019 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-972b5684-cf70-4d31-8e9f-6497a169ee41 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:37.253616 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972b5684-cf70-4d31-8e9f-6497a169ee41 service cinder] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:37.253934 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972b5684-cf70-4d31-8e9f-6497a169ee41 service cinder] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:37.254227 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972b5684-cf70-4d31-8e9f-6497a169ee41 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:37.315344 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-972b5684-cf70-4d31-8e9f-6497a169ee41 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:37.316692 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-972b5684-cf70-4d31-8e9f-6497a169ee41 service cinder] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:37.326106 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1193/2391] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:08:37 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:08:37.881878 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0119c96e-f48d-4a90-a9d8-1b727973d389 service cinder] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:37.891120 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0119c96e-f48d-4a90-a9d8-1b727973d389 service cinder] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:37.894672 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-540e3025-b565-4fb7-8a64-dfbde4579ec0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:37.895259 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540e3025-b565-4fb7-8a64-dfbde4579ec0 service cinder] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:37.895532 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540e3025-b565-4fb7-8a64-dfbde4579ec0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:37.895799 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540e3025-b565-4fb7-8a64-dfbde4579ec0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:37.905179 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-540e3025-b565-4fb7-8a64-dfbde4579ec0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:37.906662 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-540e3025-b565-4fb7-8a64-dfbde4579ec0 service cinder] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:37.916179 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1199/2392] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:08:37 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:08:38.225512 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-972b5684-cf70-4d31-8e9f-6497a169ee41 service cinder] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:38.226935 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-972b5684-cf70-4d31-8e9f-6497a169ee41 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:38.294963 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-972b5684-cf70-4d31-8e9f-6497a169ee41 service cinder] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:38.299526 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1d6ba1-9562-47bd-a7ed-68da3f77a6d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:38.300155 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1d6ba1-9562-47bd-a7ed-68da3f77a6d4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:38.300433 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1d6ba1-9562-47bd-a7ed-68da3f77a6d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:38.300701 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1d6ba1-9562-47bd-a7ed-68da3f77a6d4 admin admin] PATH_INFO: `/v3/users/d698db91fcfd4a42a552f03b14b16e52` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:38.301817 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca1d6ba1-9562-47bd-a7ed-68da3f77a6d4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d698db91fcfd4a42a552f03b14b16e52)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:38.303770 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca1d6ba1-9562-47bd-a7ed-68da3f77a6d4 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:38.333097 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:38.354314 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-ca1d6ba1-9562-47bd-a7ed-68da3f77a6d4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd698db91fcfd4a42a552f03b14b16e52'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:38.362275 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-ca1d6ba1-9562-47bd-a7ed-68da3f77a6d4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd698db91fcfd4a42a552f03b14b16e52'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:38.374489 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-ca1d6ba1-9562-47bd-a7ed-68da3f77a6d4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd698db91fcfd4a42a552f03b14b16e52'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:38.381289 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-ca1d6ba1-9562-47bd-a7ed-68da3f77a6d4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd698db91fcfd4a42a552f03b14b16e52'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:38.382091 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-ca1d6ba1-9562-47bd-a7ed-68da3f77a6d4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd698db91fcfd4a42a552f03b14b16e52'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:38.388588 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1194/2393] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:08:38 2022] DELETE /identity/v3/users/d698db91fcfd4a42a552f03b14b16e52 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:08:38.394618 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29599f80-b259-4297-ba9a-7084c4ff3f51 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:38.394903 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29599f80-b259-4297-ba9a-7084c4ff3f51 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:38.395175 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29599f80-b259-4297-ba9a-7084c4ff3f51 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:38.414647 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-29599f80-b259-4297-ba9a-7084c4ff3f51 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:08:38.415951 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-29599f80-b259-4297-ba9a-7084c4ff3f51 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:38.476562 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1200/2394] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:08:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:08:38.487364 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1d6ba1-9562-47bd-a7ed-68da3f77a6d4 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:38.488918 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-ca1d6ba1-9562-47bd-a7ed-68da3f77a6d4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:38.546859 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1d6ba1-9562-47bd-a7ed-68da3f77a6d4 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:38.550574 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f0caeb-aaa4-4a39-bb8a-9f4e3a8aa826 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:38.551193 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f0caeb-aaa4-4a39-bb8a-9f4e3a8aa826 service neutron] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:38.551464 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f0caeb-aaa4-4a39-bb8a-9f4e3a8aa826 service neutron] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:38.551729 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f0caeb-aaa4-4a39-bb8a-9f4e3a8aa826 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:38.562136 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5f0caeb-aaa4-4a39-bb8a-9f4e3a8aa826 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:38.563577 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5f0caeb-aaa4-4a39-bb8a-9f4e3a8aa826 service neutron] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:38.572831 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1195/2395] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:08:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:08:38.848532 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29599f80-b259-4297-ba9a-7084c4ff3f51 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:38.850047 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-29599f80-b259-4297-ba9a-7084c4ff3f51 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:38.862647 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29599f80-b259-4297-ba9a-7084c4ff3f51 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:38.867202 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a03edb4-708a-4691-8b3e-aa7007a6ccc0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:38.867834 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a03edb4-708a-4691-8b3e-aa7007a6ccc0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:38.868109 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a03edb4-708a-4691-8b3e-aa7007a6ccc0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:38.868377 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a03edb4-708a-4691-8b3e-aa7007a6ccc0 admin admin] PATH_INFO: `/v3/projects/6f8e62812b574b378b10d3d76ac9b43c` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:38.869506 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a03edb4-708a-4691-8b3e-aa7007a6ccc0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6f8e62812b574b378b10d3d76ac9b43c)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:38.871463 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a03edb4-708a-4691-8b3e-aa7007a6ccc0 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:38.920566 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-9a03edb4-708a-4691-8b3e-aa7007a6ccc0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6f8e62812b574b378b10d3d76ac9b43c', 'request_id': 'req-9a03edb4-708a-4691-8b3e-aa7007a6ccc0'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:38.927396 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-9a03edb4-708a-4691-8b3e-aa7007a6ccc0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6f8e62812b574b378b10d3d76ac9b43c', 'request_id': 'req-9a03edb4-708a-4691-8b3e-aa7007a6ccc0'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:38.933169 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-9a03edb4-708a-4691-8b3e-aa7007a6ccc0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6f8e62812b574b378b10d3d76ac9b43c', 'request_id': 'req-9a03edb4-708a-4691-8b3e-aa7007a6ccc0'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:38.945728 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-9a03edb4-708a-4691-8b3e-aa7007a6ccc0 admin admin] The token cache is being invalidate because project 6f8e62812b574b378b10d3d76ac9b43c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81561) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:08:38.946039 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-9a03edb4-708a-4691-8b3e-aa7007a6ccc0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:38.946703 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-9a03edb4-708a-4691-8b3e-aa7007a6ccc0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:38.948196 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1201/2396] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:08:38 2022] DELETE /identity/v3/projects/6f8e62812b574b378b10d3d76ac9b43c => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:08:38.953152 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54d4ab9-0530-4fbe-9e82-33363aeaccf2 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:38.953438 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54d4ab9-0530-4fbe-9e82-33363aeaccf2 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:38.953712 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54d4ab9-0530-4fbe-9e82-33363aeaccf2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:38.974122 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-d54d4ab9-0530-4fbe-9e82-33363aeaccf2 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:08:38.975354 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-d54d4ab9-0530-4fbe-9e82-33363aeaccf2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:39.041350 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1196/2397] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:08:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:08:39.049588 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a657c4-1ddf-4418-a096-dfdcd211fa16 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:39.049875 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a657c4-1ddf-4418-a096-dfdcd211fa16 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:39.050150 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a657c4-1ddf-4418-a096-dfdcd211fa16 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:39.071297 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-28a657c4-1ddf-4418-a096-dfdcd211fa16 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:08:39.072511 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-28a657c4-1ddf-4418-a096-dfdcd211fa16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:39.082669 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1202/2398] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:08:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:08:39.086287 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d54d4ab9-0530-4fbe-9e82-33363aeaccf2 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:39.096640 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d54d4ab9-0530-4fbe-9e82-33363aeaccf2 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:39.101276 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7562a9b8-06d7-4d9d-8871-eb5a5150ea79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:39.101865 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7562a9b8-06d7-4d9d-8871-eb5a5150ea79 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:39.102560 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7562a9b8-06d7-4d9d-8871-eb5a5150ea79 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:39.102829 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7562a9b8-06d7-4d9d-8871-eb5a5150ea79 admin admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:39.103852 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7562a9b8-06d7-4d9d-8871-eb5a5150ea79 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 09:08:39.104680 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7562a9b8-06d7-4d9d-8871-eb5a5150ea79 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:39.107463 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7562a9b8-06d7-4d9d-8871-eb5a5150ea79 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:39.119264 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1197/2399] 10.222.0.36 () {58 vars in 1153 bytes} [Thu Oct 20 09:08:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 09:08:39.124244 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28a657c4-1ddf-4418-a096-dfdcd211fa16 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:39.133937 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28a657c4-1ddf-4418-a096-dfdcd211fa16 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:39.138635 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7c2328b-370a-4233-a299-9936518d08ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:39.139237 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c2328b-370a-4233-a299-9936518d08ed admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:39.139513 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c2328b-370a-4233-a299-9936518d08ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:39.139789 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c2328b-370a-4233-a299-9936518d08ed admin admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:39.140428 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7c2328b-370a-4233-a299-9936518d08ed admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:39.142318 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7c2328b-370a-4233-a299-9936518d08ed admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:39.170490 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1203/2400] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:08:39 2022] POST /identity/v3/projects => generated 403 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:08:39.174898 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7562a9b8-06d7-4d9d-8871-eb5a5150ea79 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:39.187167 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7562a9b8-06d7-4d9d-8871-eb5a5150ea79 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:39.220945 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:DELETE=6,keystone:SELECT=93,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:39.253636 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5462da8a-c7c6-43db-a701-6bc36de8a9c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:39.254249 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5462da8a-c7c6-43db-a701-6bc36de8a9c4 admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:39.254561 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5462da8a-c7c6-43db-a701-6bc36de8a9c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:39.254831 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5462da8a-c7c6-43db-a701-6bc36de8a9c4 admin admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:39.255484 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5462da8a-c7c6-43db-a701-6bc36de8a9c4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:39.257401 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5462da8a-c7c6-43db-a701-6bc36de8a9c4 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:39.280950 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1198/2401] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:08:39 2022] POST /identity/v3/users => generated 302 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 20 09:08:39.284998 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7c2328b-370a-4233-a299-9936518d08ed admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:39.296889 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7c2328b-370a-4233-a299-9936518d08ed admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:39.302209 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a9ca14b-e08a-423a-bc4a-2e428e3c4ea3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:39.302872 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a9ca14b-e08a-423a-bc4a-2e428e3c4ea3 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:39.303150 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a9ca14b-e08a-423a-bc4a-2e428e3c4ea3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:39.303422 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a9ca14b-e08a-423a-bc4a-2e428e3c4ea3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:39.304065 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a9ca14b-e08a-423a-bc4a-2e428e3c4ea3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:39.305811 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a9ca14b-e08a-423a-bc4a-2e428e3c4ea3 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:39.317886 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1204/2402] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:08:39 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:08:39.321819 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5462da8a-c7c6-43db-a701-6bc36de8a9c4 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:39.336818 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5462da8a-c7c6-43db-a701-6bc36de8a9c4 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:39.345942 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52c9167d-3830-4794-9662-7a77314ecbf9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:39.346740 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c9167d-3830-4794-9662-7a77314ecbf9 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:39.347105 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c9167d-3830-4794-9662-7a77314ecbf9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:39.347446 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c9167d-3830-4794-9662-7a77314ecbf9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:39.348305 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52c9167d-3830-4794-9662-7a77314ecbf9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:39.350079 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52c9167d-3830-4794-9662-7a77314ecbf9 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:39.361640 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1199/2403] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:08:39 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:08:39.365534 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a9ca14b-e08a-423a-bc4a-2e428e3c4ea3 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:39.376721 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a9ca14b-e08a-423a-bc4a-2e428e3c4ea3 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:39.383594 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c700700-6795-4da7-b609-158f7808211c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:39.384418 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c700700-6795-4da7-b609-158f7808211c admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:39.384760 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c700700-6795-4da7-b609-158f7808211c admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:39.385094 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c700700-6795-4da7-b609-158f7808211c admin admin] PATH_INFO: `/v3/projects/a65b32f37d7d4c5582ac0cb8ec95dcaa/users/402da0ba14a44ae4bd002fdb8aec43a1/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:39.405075 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c700700-6795-4da7-b609-158f7808211c admin admin] RBAC: Authorizing `identity:create_grant(project_id=a65b32f37d7d4c5582ac0cb8ec95dcaa, user_id=402da0ba14a44ae4bd002fdb8aec43a1, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:39.407857 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c700700-6795-4da7-b609-158f7808211c admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:39.416572 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:39.422674 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1205/2404] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:08:39 2022] PUT /identity/v3/projects/a65b32f37d7d4c5582ac0cb8ec95dcaa/users/402da0ba14a44ae4bd002fdb8aec43a1/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:08:39.433310 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12cc69ee-ea95-4688-832f-7abd5ce2528e None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:39.433596 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12cc69ee-ea95-4688-832f-7abd5ce2528e None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:39.433937 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12cc69ee-ea95-4688-832f-7abd5ce2528e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:39.462179 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-12cc69ee-ea95-4688-832f-7abd5ce2528e None None] MFA Rules not processed for user `402da0ba14a44ae4bd002fdb8aec43a1`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:08:39.463419 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-12cc69ee-ea95-4688-832f-7abd5ce2528e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:39.529195 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1200/2405] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:08:39 2022] POST /identity/v3/auth/tokens => generated 3280 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:08:39.533780 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c700700-6795-4da7-b609-158f7808211c admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:39.596716 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c700700-6795-4da7-b609-158f7808211c admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:39.601348 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f53af04-5fe0-4db1-bc80-5f109f863229 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:39.601948 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f53af04-5fe0-4db1-bc80-5f109f863229 admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:39.602229 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f53af04-5fe0-4db1-bc80-5f109f863229 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:39.602524 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f53af04-5fe0-4db1-bc80-5f109f863229 admin admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:39.603149 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f53af04-5fe0-4db1-bc80-5f109f863229 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:39.605056 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f53af04-5fe0-4db1-bc80-5f109f863229 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:39.630384 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1206/2406] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:08:39 2022] POST /identity/v3/projects => generated 405 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:08:39.634427 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12cc69ee-ea95-4688-832f-7abd5ce2528e None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:39.645528 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12cc69ee-ea95-4688-832f-7abd5ce2528e None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:39.701719 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be1603ae-1a8b-40bd-9536-e836865d74f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:39.702309 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be1603ae-1a8b-40bd-9536-e836865d74f2 admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:39.702616 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be1603ae-1a8b-40bd-9536-e836865d74f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:39.702882 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be1603ae-1a8b-40bd-9536-e836865d74f2 admin admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:39.703487 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be1603ae-1a8b-40bd-9536-e836865d74f2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:39.705665 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be1603ae-1a8b-40bd-9536-e836865d74f2 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:39.727081 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1201/2407] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:08:39 2022] POST /identity/v3/users => generated 309 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:08:39.731698 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f53af04-5fe0-4db1-bc80-5f109f863229 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:39.742250 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f53af04-5fe0-4db1-bc80-5f109f863229 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:39.747175 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34bfa352-faae-4e58-95e8-499a266e414a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:39.747781 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34bfa352-faae-4e58-95e8-499a266e414a admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:39.748059 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34bfa352-faae-4e58-95e8-499a266e414a admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:39.748339 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34bfa352-faae-4e58-95e8-499a266e414a admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:39.749005 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34bfa352-faae-4e58-95e8-499a266e414a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:39.750607 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34bfa352-faae-4e58-95e8-499a266e414a admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:39.761635 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1207/2408] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:08:39 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 20 09:08:39.765753 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be1603ae-1a8b-40bd-9536-e836865d74f2 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:39.776196 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be1603ae-1a8b-40bd-9536-e836865d74f2 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:39.781381 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6102a56-5674-4e2f-b825-2cb9d891ced0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:39.782030 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6102a56-5674-4e2f-b825-2cb9d891ced0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:39.782307 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6102a56-5674-4e2f-b825-2cb9d891ced0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:39.782621 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6102a56-5674-4e2f-b825-2cb9d891ced0 admin admin] PATH_INFO: `/v3/projects/0bd53fabcdc54d379fbfc422796f4d47/users/82d924614c3a4d83bf70d1ae9e93a7ac/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:39.797533 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6102a56-5674-4e2f-b825-2cb9d891ced0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0bd53fabcdc54d379fbfc422796f4d47, user_id=82d924614c3a4d83bf70d1ae9e93a7ac, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:39.800231 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6102a56-5674-4e2f-b825-2cb9d891ced0 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:39.812921 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1202/2409] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:08:39 2022] PUT /identity/v3/projects/0bd53fabcdc54d379fbfc422796f4d47/users/82d924614c3a4d83bf70d1ae9e93a7ac/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:08:39.816776 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34bfa352-faae-4e58-95e8-499a266e414a admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:39.826722 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34bfa352-faae-4e58-95e8-499a266e414a admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:39.882126 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96a1a60d-7748-4ecb-babf-8818c625fe2c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:39.882743 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96a1a60d-7748-4ecb-babf-8818c625fe2c admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:39.883018 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96a1a60d-7748-4ecb-babf-8818c625fe2c admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:39.883281 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96a1a60d-7748-4ecb-babf-8818c625fe2c admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:39.883903 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96a1a60d-7748-4ecb-babf-8818c625fe2c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:39.885485 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96a1a60d-7748-4ecb-babf-8818c625fe2c admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:39.894892 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1208/2410] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:08:39 2022] GET /identity/v3/roles => generated 1420 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:08:39.899145 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6102a56-5674-4e2f-b825-2cb9d891ced0 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:39.909737 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6102a56-5674-4e2f-b825-2cb9d891ced0 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:39.914305 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6532ee1-62d6-434f-9664-51dd443c1b6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:39.914978 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6532ee1-62d6-434f-9664-51dd443c1b6b admin admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:39.915256 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6532ee1-62d6-434f-9664-51dd443c1b6b admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:39.915527 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6532ee1-62d6-434f-9664-51dd443c1b6b admin admin] PATH_INFO: `/v3/projects/0bd53fabcdc54d379fbfc422796f4d47/users/82d924614c3a4d83bf70d1ae9e93a7ac/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:39.917449 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6532ee1-62d6-434f-9664-51dd443c1b6b admin admin] RBAC: Authorizing `identity:create_grant(project_id=0bd53fabcdc54d379fbfc422796f4d47, user_id=82d924614c3a4d83bf70d1ae9e93a7ac, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:39.920741 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6532ee1-62d6-434f-9664-51dd443c1b6b admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:39.922969 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:39.932228 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1203/2411] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:08:39 2022] PUT /identity/v3/projects/0bd53fabcdc54d379fbfc422796f4d47/users/82d924614c3a4d83bf70d1ae9e93a7ac/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:08:39.943883 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f6bb57-53b1-4bd7-9ce6-a771416d7049 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:39.944375 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f6bb57-53b1-4bd7-9ce6-a771416d7049 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:39.944503 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f6bb57-53b1-4bd7-9ce6-a771416d7049 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:39.974053 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-07f6bb57-53b1-4bd7-9ce6-a771416d7049 None None] MFA Rules not processed for user `82d924614c3a4d83bf70d1ae9e93a7ac`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:08:39.975982 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-07f6bb57-53b1-4bd7-9ce6-a771416d7049 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:40.035046 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:40.046846 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1209/2412] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:08:39 2022] POST /identity/v3/auth/tokens => generated 3349 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:08:40.057584 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6532ee1-62d6-434f-9664-51dd443c1b6b admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:40.059311 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-f6532ee1-62d6-434f-9664-51dd443c1b6b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:40.119433 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6532ee1-62d6-434f-9664-51dd443c1b6b admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:40.123078 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5092c8c5-42a1-461b-8975-c74d851f06dc service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:40.123672 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5092c8c5-42a1-461b-8975-c74d851f06dc service cinder] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:40.123947 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5092c8c5-42a1-461b-8975-c74d851f06dc service cinder] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:40.124217 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5092c8c5-42a1-461b-8975-c74d851f06dc service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:40.133019 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5092c8c5-42a1-461b-8975-c74d851f06dc service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:40.134427 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5092c8c5-42a1-461b-8975-c74d851f06dc service cinder] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:40.143640 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1204/2413] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:08:40 2022] GET /identity/v3/auth/tokens => generated 3349 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:08:40.433325 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07f6bb57-53b1-4bd7-9ce6-a771416d7049 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:40.498666 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07f6bb57-53b1-4bd7-9ce6-a771416d7049 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:40.502752 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b50508f-a8cb-405f-9ae5-9ae7a564e606 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:40.503456 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b50508f-a8cb-405f-9ae5-9ae7a564e606 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:40.503732 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b50508f-a8cb-405f-9ae5-9ae7a564e606 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:40.504107 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b50508f-a8cb-405f-9ae5-9ae7a564e606 admin admin] PATH_INFO: `/v3/users/402da0ba14a44ae4bd002fdb8aec43a1` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:40.505218 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b50508f-a8cb-405f-9ae5-9ae7a564e606 admin admin] RBAC: Authorizing `identity:delete_user(user_id=402da0ba14a44ae4bd002fdb8aec43a1)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:40.507179 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b50508f-a8cb-405f-9ae5-9ae7a564e606 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:40.555982 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-8b50508f-a8cb-405f-9ae5-9ae7a564e606 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '402da0ba14a44ae4bd002fdb8aec43a1'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:40.565590 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-8b50508f-a8cb-405f-9ae5-9ae7a564e606 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '402da0ba14a44ae4bd002fdb8aec43a1'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:40.580952 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-8b50508f-a8cb-405f-9ae5-9ae7a564e606 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '402da0ba14a44ae4bd002fdb8aec43a1'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:40.590959 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-8b50508f-a8cb-405f-9ae5-9ae7a564e606 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '402da0ba14a44ae4bd002fdb8aec43a1'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:40.591681 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-8b50508f-a8cb-405f-9ae5-9ae7a564e606 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '402da0ba14a44ae4bd002fdb8aec43a1'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:40.598284 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1210/2414] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:08:40 2022] DELETE /identity/v3/users/402da0ba14a44ae4bd002fdb8aec43a1 => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:08:40.602794 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5dc367-9e09-4c5b-a9ee-07395884c6b8 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:40.603286 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5dc367-9e09-4c5b-a9ee-07395884c6b8 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:40.603402 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5dc367-9e09-4c5b-a9ee-07395884c6b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:40.625419 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-af5dc367-9e09-4c5b-a9ee-07395884c6b8 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:08:40.626640 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-af5dc367-9e09-4c5b-a9ee-07395884c6b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:40.687611 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1205/2415] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:08:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:08:40.697892 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b50508f-a8cb-405f-9ae5-9ae7a564e606 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:40.699479 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-8b50508f-a8cb-405f-9ae5-9ae7a564e606 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:40.764454 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b50508f-a8cb-405f-9ae5-9ae7a564e606 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:40.768113 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f92a541e-5735-4845-9a2a-c33e64c6f7fc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:40.768703 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f92a541e-5735-4845-9a2a-c33e64c6f7fc service neutron] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:40.768976 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f92a541e-5735-4845-9a2a-c33e64c6f7fc service neutron] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:40.769243 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f92a541e-5735-4845-9a2a-c33e64c6f7fc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:40.777991 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f92a541e-5735-4845-9a2a-c33e64c6f7fc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:40.779520 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f92a541e-5735-4845-9a2a-c33e64c6f7fc service neutron] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:40.789058 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1211/2416] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:08:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:08:41.316876 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af5dc367-9e09-4c5b-a9ee-07395884c6b8 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:41.318461 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-af5dc367-9e09-4c5b-a9ee-07395884c6b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:41.331185 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af5dc367-9e09-4c5b-a9ee-07395884c6b8 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:41.336044 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0debe4d7-3a19-474f-83f5-0b3a3c9d3c25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:41.336706 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0debe4d7-3a19-474f-83f5-0b3a3c9d3c25 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:41.336979 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0debe4d7-3a19-474f-83f5-0b3a3c9d3c25 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:41.337275 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0debe4d7-3a19-474f-83f5-0b3a3c9d3c25 admin admin] PATH_INFO: `/v3/projects/a65b32f37d7d4c5582ac0cb8ec95dcaa` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:41.338661 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0debe4d7-3a19-474f-83f5-0b3a3c9d3c25 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a65b32f37d7d4c5582ac0cb8ec95dcaa)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:41.341517 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0debe4d7-3a19-474f-83f5-0b3a3c9d3c25 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:41.391020 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-0debe4d7-3a19-474f-83f5-0b3a3c9d3c25 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a65b32f37d7d4c5582ac0cb8ec95dcaa', 'request_id': 'req-0debe4d7-3a19-474f-83f5-0b3a3c9d3c25'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:41.397305 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-0debe4d7-3a19-474f-83f5-0b3a3c9d3c25 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a65b32f37d7d4c5582ac0cb8ec95dcaa', 'request_id': 'req-0debe4d7-3a19-474f-83f5-0b3a3c9d3c25'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:41.403367 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-0debe4d7-3a19-474f-83f5-0b3a3c9d3c25 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a65b32f37d7d4c5582ac0cb8ec95dcaa', 'request_id': 'req-0debe4d7-3a19-474f-83f5-0b3a3c9d3c25'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:41.415391 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-0debe4d7-3a19-474f-83f5-0b3a3c9d3c25 admin admin] The token cache is being invalidate because project a65b32f37d7d4c5582ac0cb8ec95dcaa was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81562) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:08:41.415738 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-0debe4d7-3a19-474f-83f5-0b3a3c9d3c25 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:41.416371 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-0debe4d7-3a19-474f-83f5-0b3a3c9d3c25 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:41.417853 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1206/2417] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:08:41 2022] DELETE /identity/v3/projects/a65b32f37d7d4c5582ac0cb8ec95dcaa => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:08:41.422349 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f92a541e-5735-4845-9a2a-c33e64c6f7fc service neutron] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:41.432283 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f92a541e-5735-4845-9a2a-c33e64c6f7fc service neutron] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:41.444222 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:41.493468 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78aa987c-05c2-4ba7-a8f4-55e0343027d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:41.494115 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78aa987c-05c2-4ba7-a8f4-55e0343027d0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:41.494393 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78aa987c-05c2-4ba7-a8f4-55e0343027d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:41.494691 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78aa987c-05c2-4ba7-a8f4-55e0343027d0 admin admin] PATH_INFO: `/v3/users/82d924614c3a4d83bf70d1ae9e93a7ac` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:41.495791 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78aa987c-05c2-4ba7-a8f4-55e0343027d0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=82d924614c3a4d83bf70d1ae9e93a7ac)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:41.497737 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78aa987c-05c2-4ba7-a8f4-55e0343027d0 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:41.544147 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-78aa987c-05c2-4ba7-a8f4-55e0343027d0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '82d924614c3a4d83bf70d1ae9e93a7ac'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:41.550605 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-78aa987c-05c2-4ba7-a8f4-55e0343027d0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '82d924614c3a4d83bf70d1ae9e93a7ac'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:41.562257 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-78aa987c-05c2-4ba7-a8f4-55e0343027d0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '82d924614c3a4d83bf70d1ae9e93a7ac'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:41.568977 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-78aa987c-05c2-4ba7-a8f4-55e0343027d0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '82d924614c3a4d83bf70d1ae9e93a7ac'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:41.569599 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-78aa987c-05c2-4ba7-a8f4-55e0343027d0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '82d924614c3a4d83bf70d1ae9e93a7ac'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:41.575302 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1212/2418] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:08:41 2022] DELETE /identity/v3/users/82d924614c3a4d83bf70d1ae9e93a7ac => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:08:41.833071 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0debe4d7-3a19-474f-83f5-0b3a3c9d3c25 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:41.834728 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-0debe4d7-3a19-474f-83f5-0b3a3c9d3c25 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:41.844135 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78aa987c-05c2-4ba7-a8f4-55e0343027d0 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:41.845621 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-78aa987c-05c2-4ba7-a8f4-55e0343027d0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:41.848485 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0debe4d7-3a19-474f-83f5-0b3a3c9d3c25 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:41.869287 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:INSERT=2,keystone:SELECT=91,keystone:DELETE=7 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:41.914372 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78aa987c-05c2-4ba7-a8f4-55e0343027d0 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:41.914654 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df3c59b6-2673-4c7a-b275-f0802272ccac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:41.915167 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df3c59b6-2673-4c7a-b275-f0802272ccac admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:41.915443 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df3c59b6-2673-4c7a-b275-f0802272ccac admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:41.915709 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df3c59b6-2673-4c7a-b275-f0802272ccac admin admin] PATH_INFO: `/v3/users/851712bf63a04931bbc6925ebb3113ef` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:41.916795 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df3c59b6-2673-4c7a-b275-f0802272ccac admin admin] RBAC: Authorizing `identity:delete_user(user_id=851712bf63a04931bbc6925ebb3113ef)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:41.918774 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df3c59b6-2673-4c7a-b275-f0802272ccac admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:41.919517 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dd9b58d-bed0-4c04-abbc-b861da633945 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:41.920138 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd9b58d-bed0-4c04-abbc-b861da633945 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:41.920414 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd9b58d-bed0-4c04-abbc-b861da633945 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:41.920680 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd9b58d-bed0-4c04-abbc-b861da633945 admin admin] PATH_INFO: `/v3/projects/0bd53fabcdc54d379fbfc422796f4d47` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:41.921816 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dd9b58d-bed0-4c04-abbc-b861da633945 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0bd53fabcdc54d379fbfc422796f4d47)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:41.923724 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dd9b58d-bed0-4c04-abbc-b861da633945 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:41.963951 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:41.965972 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-df3c59b6-2673-4c7a-b275-f0802272ccac admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '851712bf63a04931bbc6925ebb3113ef'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:41.973544 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-df3c59b6-2673-4c7a-b275-f0802272ccac admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '851712bf63a04931bbc6925ebb3113ef'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:41.975772 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-8dd9b58d-bed0-4c04-abbc-b861da633945 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0bd53fabcdc54d379fbfc422796f4d47', 'request_id': 'req-8dd9b58d-bed0-4c04-abbc-b861da633945'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:41.982768 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-8dd9b58d-bed0-4c04-abbc-b861da633945 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0bd53fabcdc54d379fbfc422796f4d47', 'request_id': 'req-8dd9b58d-bed0-4c04-abbc-b861da633945'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:41.986095 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-df3c59b6-2673-4c7a-b275-f0802272ccac admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '851712bf63a04931bbc6925ebb3113ef'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:41.988953 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-8dd9b58d-bed0-4c04-abbc-b861da633945 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0bd53fabcdc54d379fbfc422796f4d47', 'request_id': 'req-8dd9b58d-bed0-4c04-abbc-b861da633945'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:41.993342 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-df3c59b6-2673-4c7a-b275-f0802272ccac admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '851712bf63a04931bbc6925ebb3113ef'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:41.994073 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-df3c59b6-2673-4c7a-b275-f0802272ccac admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '851712bf63a04931bbc6925ebb3113ef'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:42.000171 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1207/2419] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:08:41 2022] DELETE /identity/v3/users/851712bf63a04931bbc6925ebb3113ef => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:08:42.001604 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-8dd9b58d-bed0-4c04-abbc-b861da633945 admin admin] The token cache is being invalidate because project 0bd53fabcdc54d379fbfc422796f4d47 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81561) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:08:42.001897 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-8dd9b58d-bed0-4c04-abbc-b861da633945 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:42.002591 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-8dd9b58d-bed0-4c04-abbc-b861da633945 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:42.004088 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1213/2420] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:08:41 2022] DELETE /identity/v3/projects/0bd53fabcdc54d379fbfc422796f4d47 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:08:42.004753 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ba4414-a9dd-45e3-be02-53a6e15d5510 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:42.005047 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ba4414-a9dd-45e3-be02-53a6e15d5510 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:42.005321 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ba4414-a9dd-45e3-be02-53a6e15d5510 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:42.009498 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f2985b0-406a-4c32-afb3-38efc098e062 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:42.009780 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f2985b0-406a-4c32-afb3-38efc098e062 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:42.010048 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f2985b0-406a-4c32-afb3-38efc098e062 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:42.025749 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-50ba4414-a9dd-45e3-be02-53a6e15d5510 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:08:42.027058 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-50ba4414-a9dd-45e3-be02-53a6e15d5510 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:42.030687 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-0f2985b0-406a-4c32-afb3-38efc098e062 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:08:42.031855 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-0f2985b0-406a-4c32-afb3-38efc098e062 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:42.088952 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1208/2421] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:08:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 20 09:08:42.100057 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1214/2422] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:08:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 20 09:08:42.101989 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78bbf7ae-21b8-49af-8dac-b71b4bfacc1f None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:42.102345 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78bbf7ae-21b8-49af-8dac-b71b4bfacc1f None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:42.102728 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78bbf7ae-21b8-49af-8dac-b71b4bfacc1f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:42.106083 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f2985b0-406a-4c32-afb3-38efc098e062 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:42.107499 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-0f2985b0-406a-4c32-afb3-38efc098e062 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:42.124501 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-78bbf7ae-21b8-49af-8dac-b71b4bfacc1f None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:08:42.125692 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-78bbf7ae-21b8-49af-8dac-b71b4bfacc1f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:42.133736 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1209/2423] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:08:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:08:42.138513 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78bbf7ae-21b8-49af-8dac-b71b4bfacc1f None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:42.147262 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78bbf7ae-21b8-49af-8dac-b71b4bfacc1f None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:42.153622 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de3bdadb-cff0-456d-8955-54b6a816aaa5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:42.154204 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de3bdadb-cff0-456d-8955-54b6a816aaa5 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:42.154563 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de3bdadb-cff0-456d-8955-54b6a816aaa5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:42.154792 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de3bdadb-cff0-456d-8955-54b6a816aaa5 admin admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:42.156514 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de3bdadb-cff0-456d-8955-54b6a816aaa5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 09:08:42.156819 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de3bdadb-cff0-456d-8955-54b6a816aaa5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:42.158450 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de3bdadb-cff0-456d-8955-54b6a816aaa5 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:42.166153 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f2985b0-406a-4c32-afb3-38efc098e062 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:42.170534 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5809a6a3-0d5c-4e43-89d0-e116dc15a44f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:42.171134 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5809a6a3-0d5c-4e43-89d0-e116dc15a44f service neutron] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:42.171409 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5809a6a3-0d5c-4e43-89d0-e116dc15a44f service neutron] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:42.174856 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1210/2424] 10.222.0.36 () {58 vars in 1153 bytes} [Thu Oct 20 09:08:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 09:08:42.175010 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5809a6a3-0d5c-4e43-89d0-e116dc15a44f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:42.175771 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de3bdadb-cff0-456d-8955-54b6a816aaa5 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:42.181492 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5809a6a3-0d5c-4e43-89d0-e116dc15a44f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:42.182961 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5809a6a3-0d5c-4e43-89d0-e116dc15a44f service neutron] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:42.185605 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de3bdadb-cff0-456d-8955-54b6a816aaa5 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:42.191294 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3ed7c6c-bb80-41b9-b53f-14e7705fafb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:42.192101 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3ed7c6c-bb80-41b9-b53f-14e7705fafb2 admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:42.192391 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3ed7c6c-bb80-41b9-b53f-14e7705fafb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:42.192666 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3ed7c6c-bb80-41b9-b53f-14e7705fafb2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:42.193333 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3ed7c6c-bb80-41b9-b53f-14e7705fafb2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:42.193936 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1215/2425] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:08:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 20 09:08:42.195288 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3ed7c6c-bb80-41b9-b53f-14e7705fafb2 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:42.217910 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:42.230373 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1211/2426] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:08:42 2022] POST /identity/v3/projects => generated 397 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 20 09:08:42.231216 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5809a6a3-0d5c-4e43-89d0-e116dc15a44f service neutron] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:42.241181 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5809a6a3-0d5c-4e43-89d0-e116dc15a44f service neutron] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:42.314810 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88459e91-d3a2-472f-b244-d8cd1d6aec2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:42.315859 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88459e91-d3a2-472f-b244-d8cd1d6aec2a admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:42.316493 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88459e91-d3a2-472f-b244-d8cd1d6aec2a admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:42.317177 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88459e91-d3a2-472f-b244-d8cd1d6aec2a admin admin] PATH_INFO: `/v3/users` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:42.318027 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88459e91-d3a2-472f-b244-d8cd1d6aec2a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:42.320607 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88459e91-d3a2-472f-b244-d8cd1d6aec2a admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:42.341304 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=92,keystone:DELETE=4,keystone:INSERT=4 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:42.374109 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3ed7c6c-bb80-41b9-b53f-14e7705fafb2 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:42.374342 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1216/2427] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:08:42 2022] POST /identity/v3/users => generated 299 bytes in 143 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:08:42.388501 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3ed7c6c-bb80-41b9-b53f-14e7705fafb2 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:42.393365 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b93e4fa-09b0-48f8-b75f-65b56a804805 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:42.393975 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b93e4fa-09b0-48f8-b75f-65b56a804805 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:42.394240 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b93e4fa-09b0-48f8-b75f-65b56a804805 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:42.394538 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b93e4fa-09b0-48f8-b75f-65b56a804805 admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:42.395173 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b93e4fa-09b0-48f8-b75f-65b56a804805 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:42.396759 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b93e4fa-09b0-48f8-b75f-65b56a804805 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:42.410527 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1212/2428] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:08:42 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:08:42.411987 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88459e91-d3a2-472f-b244-d8cd1d6aec2a admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:42.423886 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88459e91-d3a2-472f-b244-d8cd1d6aec2a admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:42.429654 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d4f72ee-726d-4dfa-a234-c9e0cf4778b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:42.430494 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4f72ee-726d-4dfa-a234-c9e0cf4778b0 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:42.430849 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4f72ee-726d-4dfa-a234-c9e0cf4778b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:42.431185 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4f72ee-726d-4dfa-a234-c9e0cf4778b0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:42.431911 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d4f72ee-726d-4dfa-a234-c9e0cf4778b0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:42.433559 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d4f72ee-726d-4dfa-a234-c9e0cf4778b0 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:42.448435 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1217/2429] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:08:42 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:08:42.450318 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b93e4fa-09b0-48f8-b75f-65b56a804805 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:42.460878 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b93e4fa-09b0-48f8-b75f-65b56a804805 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:42.467311 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-124dec9b-e9b6-4203-9fbc-d2e071ef237a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:42.468050 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-124dec9b-e9b6-4203-9fbc-d2e071ef237a admin admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:42.468703 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-124dec9b-e9b6-4203-9fbc-d2e071ef237a admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:42.471627 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-124dec9b-e9b6-4203-9fbc-d2e071ef237a admin admin] PATH_INFO: `/v3/projects/73148d6d2be14453b3614b4276672ca1/users/fd615f9fd25c4602a4609a1444f725fa/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:42.489355 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-124dec9b-e9b6-4203-9fbc-d2e071ef237a admin admin] RBAC: Authorizing `identity:create_grant(project_id=73148d6d2be14453b3614b4276672ca1, user_id=fd615f9fd25c4602a4609a1444f725fa, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:42.492099 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-124dec9b-e9b6-4203-9fbc-d2e071ef237a admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:42.515344 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1213/2430] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:08:42 2022] PUT /identity/v3/projects/73148d6d2be14453b3614b4276672ca1/users/fd615f9fd25c4602a4609a1444f725fa/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:08:42.524964 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d4f72ee-726d-4dfa-a234-c9e0cf4778b0 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:42.526953 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-3d4f72ee-726d-4dfa-a234-c9e0cf4778b0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:42.532805 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37dfa90e-f6e6-4bad-93a8-31422177e408 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:42.533090 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37dfa90e-f6e6-4bad-93a8-31422177e408 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:42.533365 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37dfa90e-f6e6-4bad-93a8-31422177e408 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:42.544106 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d4f72ee-726d-4dfa-a234-c9e0cf4778b0 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:42.571382 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-37dfa90e-f6e6-4bad-93a8-31422177e408 None None] MFA Rules not processed for user `fd615f9fd25c4602a4609a1444f725fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:08:42.572609 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-37dfa90e-f6e6-4bad-93a8-31422177e408 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:42.612317 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf0077a8-b743-4e90-a326-568d3cfcbc17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:42.612958 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf0077a8-b743-4e90-a326-568d3cfcbc17 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:42.613241 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf0077a8-b743-4e90-a326-568d3cfcbc17 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:42.613575 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf0077a8-b743-4e90-a326-568d3cfcbc17 admin admin] PATH_INFO: `/v3/projects/cdaedc6e2cd44c3a82ce641847fff8d9` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:42.614941 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf0077a8-b743-4e90-a326-568d3cfcbc17 admin admin] RBAC: Authorizing `identity:delete_project(project_id=cdaedc6e2cd44c3a82ce641847fff8d9)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:42.616893 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf0077a8-b743-4e90-a326-568d3cfcbc17 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:42.645433 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1214/2431] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:08:42 2022] POST /identity/v3/auth/tokens => generated 3274 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:08:42.650124 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37dfa90e-f6e6-4bad-93a8-31422177e408 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:42.660231 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37dfa90e-f6e6-4bad-93a8-31422177e408 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:42.669172 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-bf0077a8-b743-4e90-a326-568d3cfcbc17 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cdaedc6e2cd44c3a82ce641847fff8d9', 'request_id': 'req-bf0077a8-b743-4e90-a326-568d3cfcbc17'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:42.675888 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-bf0077a8-b743-4e90-a326-568d3cfcbc17 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cdaedc6e2cd44c3a82ce641847fff8d9', 'request_id': 'req-bf0077a8-b743-4e90-a326-568d3cfcbc17'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:42.681867 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-bf0077a8-b743-4e90-a326-568d3cfcbc17 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cdaedc6e2cd44c3a82ce641847fff8d9', 'request_id': 'req-bf0077a8-b743-4e90-a326-568d3cfcbc17'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:42.696057 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-bf0077a8-b743-4e90-a326-568d3cfcbc17 admin admin] The token cache is being invalidate because project cdaedc6e2cd44c3a82ce641847fff8d9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81561) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:08:42.696339 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-bf0077a8-b743-4e90-a326-568d3cfcbc17 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:42.696975 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-bf0077a8-b743-4e90-a326-568d3cfcbc17 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:42.701698 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1218/2432] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:08:42 2022] DELETE /identity/v3/projects/cdaedc6e2cd44c3a82ce641847fff8d9 => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:08:42.702803 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf0077a8-b743-4e90-a326-568d3cfcbc17 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:42.704262 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-bf0077a8-b743-4e90-a326-568d3cfcbc17 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:42.736575 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9dd02d3-b8c9-4e35-bfb6-5ecf4615aeaf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:42.737399 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9dd02d3-b8c9-4e35-bfb6-5ecf4615aeaf admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:42.737753 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9dd02d3-b8c9-4e35-bfb6-5ecf4615aeaf admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:42.738089 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9dd02d3-b8c9-4e35-bfb6-5ecf4615aeaf admin admin] PATH_INFO: `/v3/projects` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:42.739031 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9dd02d3-b8c9-4e35-bfb6-5ecf4615aeaf admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:42.741072 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9dd02d3-b8c9-4e35-bfb6-5ecf4615aeaf admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:42.771950 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1215/2433] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:08:42 2022] POST /identity/v3/projects => generated 401 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 20 09:08:42.773019 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9dd02d3-b8c9-4e35-bfb6-5ecf4615aeaf admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:42.774638 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-e9dd02d3-b8c9-4e35-bfb6-5ecf4615aeaf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:42.776324 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf0077a8-b743-4e90-a326-568d3cfcbc17 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:42.796411 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:42.815982 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:42.835392 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2815c33-e770-42aa-8193-9185aa03322c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:42.835887 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2815c33-e770-42aa-8193-9185aa03322c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:42.836162 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2815c33-e770-42aa-8193-9185aa03322c admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:42.836436 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2815c33-e770-42aa-8193-9185aa03322c admin admin] PATH_INFO: `/v3/users/487c94fdc1b94de7bda88116203924f7` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:42.837564 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2815c33-e770-42aa-8193-9185aa03322c admin admin] RBAC: Authorizing `identity:delete_user(user_id=487c94fdc1b94de7bda88116203924f7)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:42.839510 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2815c33-e770-42aa-8193-9185aa03322c admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:42.840834 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9dd02d3-b8c9-4e35-bfb6-5ecf4615aeaf admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:42.845470 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecb26362-b6b7-418f-8d71-aa1e0b75e9f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:42.846091 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb26362-b6b7-418f-8d71-aa1e0b75e9f1 admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:42.846391 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb26362-b6b7-418f-8d71-aa1e0b75e9f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:42.846692 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb26362-b6b7-418f-8d71-aa1e0b75e9f1 admin admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:42.847313 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecb26362-b6b7-418f-8d71-aa1e0b75e9f1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:42.849211 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecb26362-b6b7-418f-8d71-aa1e0b75e9f1 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:42.872555 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1216/2434] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:08:42 2022] POST /identity/v3/users => generated 307 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 20 09:08:42.876741 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecb26362-b6b7-418f-8d71-aa1e0b75e9f1 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:42.886310 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecb26362-b6b7-418f-8d71-aa1e0b75e9f1 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:42.889484 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-a2815c33-e770-42aa-8193-9185aa03322c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '487c94fdc1b94de7bda88116203924f7'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:42.891970 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01815452-34f2-4b0c-9ff1-871f95887a90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:42.892747 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01815452-34f2-4b0c-9ff1-871f95887a90 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:42.893021 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01815452-34f2-4b0c-9ff1-871f95887a90 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:42.893289 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01815452-34f2-4b0c-9ff1-871f95887a90 admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:42.893956 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01815452-34f2-4b0c-9ff1-871f95887a90 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:42.896058 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01815452-34f2-4b0c-9ff1-871f95887a90 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:42.897621 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-a2815c33-e770-42aa-8193-9185aa03322c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '487c94fdc1b94de7bda88116203924f7'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:42.909990 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1217/2435] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:08:42 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 20 09:08:42.910170 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-a2815c33-e770-42aa-8193-9185aa03322c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '487c94fdc1b94de7bda88116203924f7'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:42.910981 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01815452-34f2-4b0c-9ff1-871f95887a90 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:42.916634 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-a2815c33-e770-42aa-8193-9185aa03322c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '487c94fdc1b94de7bda88116203924f7'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:42.917324 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-a2815c33-e770-42aa-8193-9185aa03322c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '487c94fdc1b94de7bda88116203924f7'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:42.920399 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01815452-34f2-4b0c-9ff1-871f95887a90 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:42.923605 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1219/2436] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:08:42 2022] DELETE /identity/v3/users/487c94fdc1b94de7bda88116203924f7 => generated 0 bytes in 221 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:08:42.925306 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a53470-4277-4c9a-9d71-ac3bbbec409b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:42.926008 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a53470-4277-4c9a-9d71-ac3bbbec409b admin admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:42.926434 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a53470-4277-4c9a-9d71-ac3bbbec409b admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:42.926739 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a53470-4277-4c9a-9d71-ac3bbbec409b admin admin] PATH_INFO: `/v3/projects/de1e4f0f8d59429db905008319c10a31/users/bed4c297f1074bd8b71c906f8d17f79f/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:42.941976 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94a53470-4277-4c9a-9d71-ac3bbbec409b admin admin] RBAC: Authorizing `identity:create_grant(project_id=de1e4f0f8d59429db905008319c10a31, user_id=bed4c297f1074bd8b71c906f8d17f79f, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:42.944677 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94a53470-4277-4c9a-9d71-ac3bbbec409b admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:42.958054 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1218/2437] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:08:42 2022] PUT /identity/v3/projects/de1e4f0f8d59429db905008319c10a31/users/bed4c297f1074bd8b71c906f8d17f79f/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:08:42.962016 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2815c33-e770-42aa-8193-9185aa03322c admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:42.963703 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-a2815c33-e770-42aa-8193-9185aa03322c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:43.034421 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2815c33-e770-42aa-8193-9185aa03322c admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:43.085223 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:43.098248 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7d1dc1a-4a23-40d5-bb5d-b0646c09585a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:43.098893 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d1dc1a-4a23-40d5-bb5d-b0646c09585a admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:43.099169 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d1dc1a-4a23-40d5-bb5d-b0646c09585a admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:43.099440 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d1dc1a-4a23-40d5-bb5d-b0646c09585a admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:43.100234 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7d1dc1a-4a23-40d5-bb5d-b0646c09585a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:43.101820 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7d1dc1a-4a23-40d5-bb5d-b0646c09585a admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:43.112043 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1220/2438] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:08:42 2022] GET /identity/v3/roles => generated 1420 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:08:43.116452 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a53470-4277-4c9a-9d71-ac3bbbec409b admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:43.127768 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a53470-4277-4c9a-9d71-ac3bbbec409b admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:43.132370 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5532d1fa-e15a-4853-b7d9-109aa74bfb3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:43.133016 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5532d1fa-e15a-4853-b7d9-109aa74bfb3e admin admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:43.133295 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5532d1fa-e15a-4853-b7d9-109aa74bfb3e admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:43.133564 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5532d1fa-e15a-4853-b7d9-109aa74bfb3e admin admin] PATH_INFO: `/v3/projects/de1e4f0f8d59429db905008319c10a31/users/bed4c297f1074bd8b71c906f8d17f79f/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:43.135542 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5532d1fa-e15a-4853-b7d9-109aa74bfb3e admin admin] RBAC: Authorizing `identity:create_grant(project_id=de1e4f0f8d59429db905008319c10a31, user_id=bed4c297f1074bd8b71c906f8d17f79f, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:43.138784 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5532d1fa-e15a-4853-b7d9-109aa74bfb3e admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:43.149851 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1219/2439] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:08:43 2022] PUT /identity/v3/projects/de1e4f0f8d59429db905008319c10a31/users/bed4c297f1074bd8b71c906f8d17f79f/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:08:43.162080 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64692e2f-8c5d-4a35-a9aa-774941b74643 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:43.162368 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64692e2f-8c5d-4a35-a9aa-774941b74643 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:43.162670 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64692e2f-8c5d-4a35-a9aa-774941b74643 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:43.192686 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-64692e2f-8c5d-4a35-a9aa-774941b74643 None None] MFA Rules not processed for user `bed4c297f1074bd8b71c906f8d17f79f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:08:43.194020 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-64692e2f-8c5d-4a35-a9aa-774941b74643 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:43.203651 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5532d1fa-e15a-4853-b7d9-109aa74bfb3e admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:43.205531 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-5532d1fa-e15a-4853-b7d9-109aa74bfb3e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:43.273588 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1221/2440] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:08:43 2022] POST /identity/v3/auth/tokens => generated 3345 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:08:43.281567 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5532d1fa-e15a-4853-b7d9-109aa74bfb3e admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:43.284769 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64692e2f-8c5d-4a35-a9aa-774941b74643 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:43.286084 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-64692e2f-8c5d-4a35-a9aa-774941b74643 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:43.286465 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e37574e-b573-4e53-8902-f2dc2b419ed6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:43.287236 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e37574e-b573-4e53-8902-f2dc2b419ed6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:43.287493 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e37574e-b573-4e53-8902-f2dc2b419ed6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:43.287769 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e37574e-b573-4e53-8902-f2dc2b419ed6 admin admin] PATH_INFO: `/v3/projects/bb7baa8bfcbc49648399cac1bf51d820` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:43.288927 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e37574e-b573-4e53-8902-f2dc2b419ed6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bb7baa8bfcbc49648399cac1bf51d820)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:43.290901 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e37574e-b573-4e53-8902-f2dc2b419ed6 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:43.338960 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-8e37574e-b573-4e53-8902-f2dc2b419ed6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bb7baa8bfcbc49648399cac1bf51d820', 'request_id': 'req-8e37574e-b573-4e53-8902-f2dc2b419ed6'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:43.344175 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64692e2f-8c5d-4a35-a9aa-774941b74643 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:43.347182 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-8e37574e-b573-4e53-8902-f2dc2b419ed6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bb7baa8bfcbc49648399cac1bf51d820', 'request_id': 'req-8e37574e-b573-4e53-8902-f2dc2b419ed6'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:43.353196 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-8e37574e-b573-4e53-8902-f2dc2b419ed6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bb7baa8bfcbc49648399cac1bf51d820', 'request_id': 'req-8e37574e-b573-4e53-8902-f2dc2b419ed6'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:43.365813 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-8e37574e-b573-4e53-8902-f2dc2b419ed6 admin admin] The token cache is being invalidate because project bb7baa8bfcbc49648399cac1bf51d820 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81562) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:08:43.366104 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-8e37574e-b573-4e53-8902-f2dc2b419ed6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:43.366821 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-8e37574e-b573-4e53-8902-f2dc2b419ed6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:43.371414 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1220/2441] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:08:43 2022] DELETE /identity/v3/projects/bb7baa8bfcbc49648399cac1bf51d820 => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:08:43.372287 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e37574e-b573-4e53-8902-f2dc2b419ed6 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:43.373666 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-8e37574e-b573-4e53-8902-f2dc2b419ed6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:43.385494 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=87,keystone:INSERT=6,keystone:DELETE=7 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:43.399454 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-756b10c7-2c7b-420b-99c3-4f5634309d70 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:43.400176 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-756b10c7-2c7b-420b-99c3-4f5634309d70 service cinder] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:43.400456 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-756b10c7-2c7b-420b-99c3-4f5634309d70 service cinder] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:43.400727 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-756b10c7-2c7b-420b-99c3-4f5634309d70 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:43.403122 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-756b10c7-2c7b-420b-99c3-4f5634309d70 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:43.442352 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e37574e-b573-4e53-8902-f2dc2b419ed6 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:43.446668 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:43.447262 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd184e1-6282-4ebf-b80a-c6894ef29a45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:43.447608 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd184e1-6282-4ebf-b80a-c6894ef29a45 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:43.447883 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd184e1-6282-4ebf-b80a-c6894ef29a45 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:43.448153 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd184e1-6282-4ebf-b80a-c6894ef29a45 admin admin] PATH_INFO: `/v3/users/5dd5dab6e7014daf8db03a631958ebd1` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:43.449982 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcd184e1-6282-4ebf-b80a-c6894ef29a45 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5dd5dab6e7014daf8db03a631958ebd1)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:43.452007 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcd184e1-6282-4ebf-b80a-c6894ef29a45 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:43.476451 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-756b10c7-2c7b-420b-99c3-4f5634309d70 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:43.477944 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-756b10c7-2c7b-420b-99c3-4f5634309d70 service cinder] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:43.494692 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1222/2442] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:08:43 2022] GET /identity/v3/auth/tokens => generated 3345 bytes in 210 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 20 09:08:43.507952 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-bcd184e1-6282-4ebf-b80a-c6894ef29a45 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5dd5dab6e7014daf8db03a631958ebd1'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:43.516465 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-bcd184e1-6282-4ebf-b80a-c6894ef29a45 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5dd5dab6e7014daf8db03a631958ebd1'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:43.532905 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-bcd184e1-6282-4ebf-b80a-c6894ef29a45 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5dd5dab6e7014daf8db03a631958ebd1'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:43.540408 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-bcd184e1-6282-4ebf-b80a-c6894ef29a45 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5dd5dab6e7014daf8db03a631958ebd1'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:43.541156 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-bcd184e1-6282-4ebf-b80a-c6894ef29a45 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5dd5dab6e7014daf8db03a631958ebd1'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:43.549165 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1221/2443] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:08:43 2022] DELETE /identity/v3/users/5dd5dab6e7014daf8db03a631958ebd1 => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:08:43.573648 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-756b10c7-2c7b-420b-99c3-4f5634309d70 service cinder] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:43.575068 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-756b10c7-2c7b-420b-99c3-4f5634309d70 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:43.586198 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-756b10c7-2c7b-420b-99c3-4f5634309d70 service cinder] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:43.651124 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a28ca68-92d3-47b5-9737-c552a0c7ebc9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:43.651826 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a28ca68-92d3-47b5-9737-c552a0c7ebc9 service cinder] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:43.652007 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a28ca68-92d3-47b5-9737-c552a0c7ebc9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:43.652255 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a28ca68-92d3-47b5-9737-c552a0c7ebc9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:43.655092 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-2a28ca68-92d3-47b5-9737-c552a0c7ebc9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:43.724437 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a28ca68-92d3-47b5-9737-c552a0c7ebc9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:43.725993 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a28ca68-92d3-47b5-9737-c552a0c7ebc9 service cinder] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:43.736732 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1223/2444] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:08:43 2022] GET /identity/v3/auth/tokens => generated 3274 bytes in 163 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:08:43.865194 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd184e1-6282-4ebf-b80a-c6894ef29a45 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:43.868403 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-bcd184e1-6282-4ebf-b80a-c6894ef29a45 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:43.944827 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd184e1-6282-4ebf-b80a-c6894ef29a45 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:43.951066 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c0eb38e-fbab-4328-8f0a-42c9315c585a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:43.951764 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0eb38e-fbab-4328-8f0a-42c9315c585a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:43.952125 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0eb38e-fbab-4328-8f0a-42c9315c585a admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:43.952804 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0eb38e-fbab-4328-8f0a-42c9315c585a admin admin] PATH_INFO: `/v3/projects/3a67e0084bba4195b8e721ffc451b793` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:43.953689 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c0eb38e-fbab-4328-8f0a-42c9315c585a admin admin] RBAC: Authorizing `identity:delete_project(project_id=3a67e0084bba4195b8e721ffc451b793)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:43.955680 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c0eb38e-fbab-4328-8f0a-42c9315c585a admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:43.997965 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:44.015267 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-0c0eb38e-fbab-4328-8f0a-42c9315c585a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3a67e0084bba4195b8e721ffc451b793', 'request_id': 'req-0c0eb38e-fbab-4328-8f0a-42c9315c585a'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:44.023604 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-0c0eb38e-fbab-4328-8f0a-42c9315c585a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3a67e0084bba4195b8e721ffc451b793', 'request_id': 'req-0c0eb38e-fbab-4328-8f0a-42c9315c585a'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:44.031681 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-0c0eb38e-fbab-4328-8f0a-42c9315c585a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3a67e0084bba4195b8e721ffc451b793', 'request_id': 'req-0c0eb38e-fbab-4328-8f0a-42c9315c585a'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:44.050093 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-0c0eb38e-fbab-4328-8f0a-42c9315c585a admin admin] The token cache is being invalidate because project 3a67e0084bba4195b8e721ffc451b793 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81562) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:08:44.050540 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-0c0eb38e-fbab-4328-8f0a-42c9315c585a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:44.051885 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-0c0eb38e-fbab-4328-8f0a-42c9315c585a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:44.054191 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1222/2445] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:08:43 2022] DELETE /identity/v3/projects/3a67e0084bba4195b8e721ffc451b793 => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:08:44.059606 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87df5f0c-9e0b-49c6-b0d8-d8b03f8bedb9 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:44.059891 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87df5f0c-9e0b-49c6-b0d8-d8b03f8bedb9 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:44.060553 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87df5f0c-9e0b-49c6-b0d8-d8b03f8bedb9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:44.095744 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-87df5f0c-9e0b-49c6-b0d8-d8b03f8bedb9 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:08:44.097573 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-87df5f0c-9e0b-49c6-b0d8-d8b03f8bedb9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:44.170319 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1224/2446] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:08:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:08:44.177809 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-656df9f7-f667-4462-8e79-17bb0c90901f None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:44.178093 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-656df9f7-f667-4462-8e79-17bb0c90901f None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:44.178356 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-656df9f7-f667-4462-8e79-17bb0c90901f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:44.198694 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-656df9f7-f667-4462-8e79-17bb0c90901f None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:08:44.200584 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-656df9f7-f667-4462-8e79-17bb0c90901f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:44.209130 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1223/2447] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:08:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:08:44.213376 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87df5f0c-9e0b-49c6-b0d8-d8b03f8bedb9 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:44.224481 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87df5f0c-9e0b-49c6-b0d8-d8b03f8bedb9 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:44.229161 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5f0cded-5cf6-4011-bd85-2a8ca3d49d13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:44.229791 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5f0cded-5cf6-4011-bd85-2a8ca3d49d13 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:44.230066 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5f0cded-5cf6-4011-bd85-2a8ca3d49d13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:44.230335 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5f0cded-5cf6-4011-bd85-2a8ca3d49d13 admin admin] PATH_INFO: `/v3/domains` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:44.231144 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5f0cded-5cf6-4011-bd85-2a8ca3d49d13 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 09:08:44.231447 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5f0cded-5cf6-4011-bd85-2a8ca3d49d13 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:44.233079 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5f0cded-5cf6-4011-bd85-2a8ca3d49d13 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:44.245334 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1225/2448] 10.222.0.36 () {58 vars in 1153 bytes} [Thu Oct 20 09:08:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 09:08:44.249765 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-656df9f7-f667-4462-8e79-17bb0c90901f None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:44.258885 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-656df9f7-f667-4462-8e79-17bb0c90901f None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:44.264274 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34f67b15-cbf7-4d22-800b-455f9fb2df0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:44.264870 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f67b15-cbf7-4d22-800b-455f9fb2df0d admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:44.265144 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f67b15-cbf7-4d22-800b-455f9fb2df0d admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:44.265409 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f67b15-cbf7-4d22-800b-455f9fb2df0d admin admin] PATH_INFO: `/v3/projects` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:44.266042 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34f67b15-cbf7-4d22-800b-455f9fb2df0d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:44.268111 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34f67b15-cbf7-4d22-800b-455f9fb2df0d admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:44.296290 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1224/2449] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:08:44 2022] POST /identity/v3/projects => generated 385 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 20 09:08:44.300398 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5f0cded-5cf6-4011-bd85-2a8ca3d49d13 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:44.310210 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5f0cded-5cf6-4011-bd85-2a8ca3d49d13 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:44.327932 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:44.368485 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a771972-bc17-4867-a663-8d3c6ee6da62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:44.369093 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a771972-bc17-4867-a663-8d3c6ee6da62 admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:44.369372 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a771972-bc17-4867-a663-8d3c6ee6da62 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:44.369642 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a771972-bc17-4867-a663-8d3c6ee6da62 admin admin] PATH_INFO: `/v3/users` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:44.370258 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a771972-bc17-4867-a663-8d3c6ee6da62 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:44.372989 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a771972-bc17-4867-a663-8d3c6ee6da62 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:44.394771 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1226/2450] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:08:44 2022] POST /identity/v3/users => generated 293 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:08:44.398881 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34f67b15-cbf7-4d22-800b-455f9fb2df0d admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:44.409617 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34f67b15-cbf7-4d22-800b-455f9fb2df0d admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:44.414303 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d523de7-2882-4276-820a-89a93ee2aba1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:44.414928 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d523de7-2882-4276-820a-89a93ee2aba1 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:44.415206 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d523de7-2882-4276-820a-89a93ee2aba1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:44.415503 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d523de7-2882-4276-820a-89a93ee2aba1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:44.416136 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d523de7-2882-4276-820a-89a93ee2aba1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:44.417788 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d523de7-2882-4276-820a-89a93ee2aba1 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:44.428259 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1225/2451] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:08:44 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:08:44.432246 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a771972-bc17-4867-a663-8d3c6ee6da62 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:44.442147 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a771972-bc17-4867-a663-8d3c6ee6da62 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:44.447000 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc33bf9f-d767-4f94-acaf-6476c8908763 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:44.447690 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc33bf9f-d767-4f94-acaf-6476c8908763 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:44.447855 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc33bf9f-d767-4f94-acaf-6476c8908763 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:44.448137 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc33bf9f-d767-4f94-acaf-6476c8908763 admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:44.448769 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc33bf9f-d767-4f94-acaf-6476c8908763 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:44.450375 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc33bf9f-d767-4f94-acaf-6476c8908763 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:44.460955 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1227/2452] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:08:44 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:08:44.465975 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d523de7-2882-4276-820a-89a93ee2aba1 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:44.475635 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d523de7-2882-4276-820a-89a93ee2aba1 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:44.480275 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a2b7b93-81c2-4734-bb5b-960bff262ec6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:44.481547 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a2b7b93-81c2-4734-bb5b-960bff262ec6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:44.481818 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a2b7b93-81c2-4734-bb5b-960bff262ec6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:44.482115 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a2b7b93-81c2-4734-bb5b-960bff262ec6 admin admin] PATH_INFO: `/v3/projects/8cd112a3910a4bb49f171cd76c65d9e5/users/c483b6f1a50d4ab0bdc57669fa22ef11/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:44.497402 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a2b7b93-81c2-4734-bb5b-960bff262ec6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8cd112a3910a4bb49f171cd76c65d9e5, user_id=c483b6f1a50d4ab0bdc57669fa22ef11, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:44.500093 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a2b7b93-81c2-4734-bb5b-960bff262ec6 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:44.512567 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1226/2453] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:08:44 2022] PUT /identity/v3/projects/8cd112a3910a4bb49f171cd76c65d9e5/users/c483b6f1a50d4ab0bdc57669fa22ef11/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:08:44.524566 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f89f9c0-9d27-4835-a2f1-9b90c4a1f61a None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:44.524855 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f89f9c0-9d27-4835-a2f1-9b90c4a1f61a None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:44.525132 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f89f9c0-9d27-4835-a2f1-9b90c4a1f61a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:44.553566 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-1f89f9c0-9d27-4835-a2f1-9b90c4a1f61a None None] MFA Rules not processed for user `c483b6f1a50d4ab0bdc57669fa22ef11`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:08:44.554742 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-1f89f9c0-9d27-4835-a2f1-9b90c4a1f61a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:44.618584 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1228/2454] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:08:44 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:08:44.630192 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a2b7b93-81c2-4734-bb5b-960bff262ec6 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:44.632565 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-5a2b7b93-81c2-4734-bb5b-960bff262ec6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:44.690412 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a2b7b93-81c2-4734-bb5b-960bff262ec6 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:44.694049 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fa06097-874a-4e41-bef4-d271f2058c44 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:44.694702 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa06097-874a-4e41-bef4-d271f2058c44 service cinder] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:44.694978 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa06097-874a-4e41-bef4-d271f2058c44 service cinder] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:44.695270 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa06097-874a-4e41-bef4-d271f2058c44 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:44.705480 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fa06097-874a-4e41-bef4-d271f2058c44 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:44.706985 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fa06097-874a-4e41-bef4-d271f2058c44 service cinder] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:44.716514 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1227/2455] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:08:44 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:08:46.236767 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f89f9c0-9d27-4835-a2f1-9b90c4a1f61a None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:46.238353 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-1f89f9c0-9d27-4835-a2f1-9b90c4a1f61a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:46.309909 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f89f9c0-9d27-4835-a2f1-9b90c4a1f61a None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:46.314192 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b4854b0-772a-4915-b589-d643f9940282 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:46.314902 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b4854b0-772a-4915-b589-d643f9940282 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:46.315178 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b4854b0-772a-4915-b589-d643f9940282 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:46.315445 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b4854b0-772a-4915-b589-d643f9940282 admin admin] PATH_INFO: `/v3/users/50abe580d021427a954401e2615714a2` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:46.316526 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b4854b0-772a-4915-b589-d643f9940282 admin admin] RBAC: Authorizing `identity:delete_user(user_id=50abe580d021427a954401e2615714a2)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:46.318464 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b4854b0-772a-4915-b589-d643f9940282 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:46.352039 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=92,keystone:INSERT=3,keystone:DELETE=5 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:46.366373 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-1b4854b0-772a-4915-b589-d643f9940282 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '50abe580d021427a954401e2615714a2'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:46.374168 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-1b4854b0-772a-4915-b589-d643f9940282 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '50abe580d021427a954401e2615714a2'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:46.386763 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-1b4854b0-772a-4915-b589-d643f9940282 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '50abe580d021427a954401e2615714a2'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:46.393571 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-1b4854b0-772a-4915-b589-d643f9940282 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '50abe580d021427a954401e2615714a2'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:46.394248 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-1b4854b0-772a-4915-b589-d643f9940282 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '50abe580d021427a954401e2615714a2'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:46.400313 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1229/2456] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:08:46 2022] DELETE /identity/v3/users/50abe580d021427a954401e2615714a2 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:08:46.404792 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8aca18a-1e5f-46e7-8c22-a72fbc456c53 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:46.405262 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8aca18a-1e5f-46e7-8c22-a72fbc456c53 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:46.405371 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8aca18a-1e5f-46e7-8c22-a72fbc456c53 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:46.424543 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-e8aca18a-1e5f-46e7-8c22-a72fbc456c53 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:08:46.425736 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-e8aca18a-1e5f-46e7-8c22-a72fbc456c53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:46.479602 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=92,keystone:DELETE=5,keystone:INSERT=3 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:46.491276 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1228/2457] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:08:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:08:46.502944 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b4854b0-772a-4915-b589-d643f9940282 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:46.504473 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-1b4854b0-772a-4915-b589-d643f9940282 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:46.567040 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b4854b0-772a-4915-b589-d643f9940282 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:46.571003 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82dac5c8-8d01-4c8d-85b4-4398d48fc344 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:46.571661 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82dac5c8-8d01-4c8d-85b4-4398d48fc344 service neutron] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:46.571961 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82dac5c8-8d01-4c8d-85b4-4398d48fc344 service neutron] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:46.572234 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82dac5c8-8d01-4c8d-85b4-4398d48fc344 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:46.581991 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82dac5c8-8d01-4c8d-85b4-4398d48fc344 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:46.583437 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82dac5c8-8d01-4c8d-85b4-4398d48fc344 service neutron] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:46.593590 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1230/2458] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:08:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:08:46.870789 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8aca18a-1e5f-46e7-8c22-a72fbc456c53 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:46.872303 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-e8aca18a-1e5f-46e7-8c22-a72fbc456c53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:46.884194 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8aca18a-1e5f-46e7-8c22-a72fbc456c53 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:46.888923 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d1076f-a0d3-438c-befa-3e325e24633a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:46.889559 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d1076f-a0d3-438c-befa-3e325e24633a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:46.889840 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d1076f-a0d3-438c-befa-3e325e24633a admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:46.890112 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d1076f-a0d3-438c-befa-3e325e24633a admin admin] PATH_INFO: `/v3/projects/89787d6388634d8c91d894dfdeccd89d` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:46.891299 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0d1076f-a0d3-438c-befa-3e325e24633a admin admin] RBAC: Authorizing `identity:delete_project(project_id=89787d6388634d8c91d894dfdeccd89d)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:46.893235 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0d1076f-a0d3-438c-befa-3e325e24633a admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:46.941380 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-d0d1076f-a0d3-438c-befa-3e325e24633a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '89787d6388634d8c91d894dfdeccd89d', 'request_id': 'req-d0d1076f-a0d3-438c-befa-3e325e24633a'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:46.947908 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-d0d1076f-a0d3-438c-befa-3e325e24633a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '89787d6388634d8c91d894dfdeccd89d', 'request_id': 'req-d0d1076f-a0d3-438c-befa-3e325e24633a'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:46.953356 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-d0d1076f-a0d3-438c-befa-3e325e24633a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '89787d6388634d8c91d894dfdeccd89d', 'request_id': 'req-d0d1076f-a0d3-438c-befa-3e325e24633a'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:46.966067 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-d0d1076f-a0d3-438c-befa-3e325e24633a admin admin] The token cache is being invalidate because project 89787d6388634d8c91d894dfdeccd89d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81562) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:08:46.966336 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-d0d1076f-a0d3-438c-befa-3e325e24633a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:46.966976 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-d0d1076f-a0d3-438c-befa-3e325e24633a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:46.968441 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1229/2459] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:08:46 2022] DELETE /identity/v3/projects/89787d6388634d8c91d894dfdeccd89d => generated 0 bytes in 98 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:08:46.973649 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75444847-1d54-4ea8-9eac-c69f2bfb3d5a None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:46.973960 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75444847-1d54-4ea8-9eac-c69f2bfb3d5a None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:46.974229 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75444847-1d54-4ea8-9eac-c69f2bfb3d5a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:46.995795 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-75444847-1d54-4ea8-9eac-c69f2bfb3d5a None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:08:46.997036 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-75444847-1d54-4ea8-9eac-c69f2bfb3d5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:47.058108 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1231/2460] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:08:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:08:47.065834 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0333879-d98c-4343-8370-de04498f0934 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:47.066102 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0333879-d98c-4343-8370-de04498f0934 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:47.066492 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0333879-d98c-4343-8370-de04498f0934 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:47.086157 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-f0333879-d98c-4343-8370-de04498f0934 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:08:47.087390 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-f0333879-d98c-4343-8370-de04498f0934 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:47.096016 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1230/2461] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:08:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:08:47.100147 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75444847-1d54-4ea8-9eac-c69f2bfb3d5a None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:47.109834 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75444847-1d54-4ea8-9eac-c69f2bfb3d5a None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:47.114443 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32ab8a0e-3b36-4632-8271-7284ee11cbe7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:47.115049 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ab8a0e-3b36-4632-8271-7284ee11cbe7 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:47.115324 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ab8a0e-3b36-4632-8271-7284ee11cbe7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:47.115590 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ab8a0e-3b36-4632-8271-7284ee11cbe7 admin admin] PATH_INFO: `/v3/domains` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:47.116252 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32ab8a0e-3b36-4632-8271-7284ee11cbe7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 09:08:47.116545 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32ab8a0e-3b36-4632-8271-7284ee11cbe7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:47.118157 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32ab8a0e-3b36-4632-8271-7284ee11cbe7 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:47.129962 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1232/2462] 10.222.0.36 () {58 vars in 1153 bytes} [Thu Oct 20 09:08:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 09:08:47.134208 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0333879-d98c-4343-8370-de04498f0934 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:47.143762 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0333879-d98c-4343-8370-de04498f0934 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:47.148372 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2de5dca1-db93-4797-a2ac-1d85bbd76dac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:47.148953 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de5dca1-db93-4797-a2ac-1d85bbd76dac admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:47.149228 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de5dca1-db93-4797-a2ac-1d85bbd76dac admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:47.149498 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de5dca1-db93-4797-a2ac-1d85bbd76dac admin admin] PATH_INFO: `/v3/projects` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:47.150113 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2de5dca1-db93-4797-a2ac-1d85bbd76dac admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:47.152032 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2de5dca1-db93-4797-a2ac-1d85bbd76dac admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:47.179768 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1231/2463] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:08:47 2022] POST /identity/v3/projects => generated 383 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:08:47.184448 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32ab8a0e-3b36-4632-8271-7284ee11cbe7 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:47.194708 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32ab8a0e-3b36-4632-8271-7284ee11cbe7 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:47.227243 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:47.253770 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00d3de63-9117-4200-8dd4-e4d6c89205b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:47.254409 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00d3de63-9117-4200-8dd4-e4d6c89205b6 admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:47.254660 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00d3de63-9117-4200-8dd4-e4d6c89205b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:47.254925 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00d3de63-9117-4200-8dd4-e4d6c89205b6 admin admin] PATH_INFO: `/v3/users` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:47.255551 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00d3de63-9117-4200-8dd4-e4d6c89205b6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:47.257462 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00d3de63-9117-4200-8dd4-e4d6c89205b6 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:47.282082 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1233/2464] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:08:47 2022] POST /identity/v3/users => generated 292 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:08:47.283350 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2de5dca1-db93-4797-a2ac-1d85bbd76dac admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:47.285147 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00d3de63-9117-4200-8dd4-e4d6c89205b6 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:47.286822 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-00d3de63-9117-4200-8dd4-e4d6c89205b6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:47.294283 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2de5dca1-db93-4797-a2ac-1d85bbd76dac admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:47.298964 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e681619-3316-47d2-8ede-01ef37c82b8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:47.299554 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e681619-3316-47d2-8ede-01ef37c82b8a admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:47.299696 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00d3de63-9117-4200-8dd4-e4d6c89205b6 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:47.299828 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e681619-3316-47d2-8ede-01ef37c82b8a admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:47.300103 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e681619-3316-47d2-8ede-01ef37c82b8a admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:47.300764 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e681619-3316-47d2-8ede-01ef37c82b8a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:47.302457 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e681619-3316-47d2-8ede-01ef37c82b8a admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:47.304225 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c072ff99-66c5-4e4c-bce2-9fb07dd2153e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:47.304839 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c072ff99-66c5-4e4c-bce2-9fb07dd2153e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:47.305117 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c072ff99-66c5-4e4c-bce2-9fb07dd2153e admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:47.305386 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c072ff99-66c5-4e4c-bce2-9fb07dd2153e admin admin] PATH_INFO: `/v3/users/c483b6f1a50d4ab0bdc57669fa22ef11` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:47.306529 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c072ff99-66c5-4e4c-bce2-9fb07dd2153e admin admin] RBAC: Authorizing `identity:delete_user(user_id=c483b6f1a50d4ab0bdc57669fa22ef11)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:47.308441 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c072ff99-66c5-4e4c-bce2-9fb07dd2153e admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:47.316123 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1232/2465] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:08:47 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:08:47.317279 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e681619-3316-47d2-8ede-01ef37c82b8a admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:47.328486 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e681619-3316-47d2-8ede-01ef37c82b8a admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:47.333448 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72f85a95-bfc3-4461-b9c5-5d5fb8833abf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:47.334066 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72f85a95-bfc3-4461-b9c5-5d5fb8833abf admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:47.334352 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72f85a95-bfc3-4461-b9c5-5d5fb8833abf admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:47.334668 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72f85a95-bfc3-4461-b9c5-5d5fb8833abf admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:47.335308 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72f85a95-bfc3-4461-b9c5-5d5fb8833abf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:47.336928 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72f85a95-bfc3-4461-b9c5-5d5fb8833abf admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:47.347796 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1233/2466] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:08:47 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:08:47.352078 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72f85a95-bfc3-4461-b9c5-5d5fb8833abf admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:47.354673 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-c072ff99-66c5-4e4c-bce2-9fb07dd2153e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c483b6f1a50d4ab0bdc57669fa22ef11'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:47.362443 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72f85a95-bfc3-4461-b9c5-5d5fb8833abf admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:47.362779 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-c072ff99-66c5-4e4c-bce2-9fb07dd2153e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c483b6f1a50d4ab0bdc57669fa22ef11'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:47.367905 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7067961c-fed9-4ca4-824a-857bf1b9779d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:47.369306 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7067961c-fed9-4ca4-824a-857bf1b9779d admin admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:47.369687 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7067961c-fed9-4ca4-824a-857bf1b9779d admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:47.370251 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7067961c-fed9-4ca4-824a-857bf1b9779d admin admin] PATH_INFO: `/v3/projects/05700a8ee8f942949117a3bf1643b00e/users/5bd3d936dcaf4fd8926e93b17b38114c/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:47.376362 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-c072ff99-66c5-4e4c-bce2-9fb07dd2153e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c483b6f1a50d4ab0bdc57669fa22ef11'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:47.383587 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-c072ff99-66c5-4e4c-bce2-9fb07dd2153e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c483b6f1a50d4ab0bdc57669fa22ef11'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:47.384337 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-c072ff99-66c5-4e4c-bce2-9fb07dd2153e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c483b6f1a50d4ab0bdc57669fa22ef11'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:47.387198 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7067961c-fed9-4ca4-824a-857bf1b9779d admin admin] RBAC: Authorizing `identity:create_grant(project_id=05700a8ee8f942949117a3bf1643b00e, user_id=5bd3d936dcaf4fd8926e93b17b38114c, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:47.389864 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7067961c-fed9-4ca4-824a-857bf1b9779d admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:47.390392 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1234/2467] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:08:47 2022] DELETE /identity/v3/users/c483b6f1a50d4ab0bdc57669fa22ef11 => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:08:47.394680 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e6e6d4-1761-4774-b26c-504b340ecc05 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:47.394965 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e6e6d4-1761-4774-b26c-504b340ecc05 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:47.395238 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e6e6d4-1761-4774-b26c-504b340ecc05 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:47.407528 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1234/2468] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:08:47 2022] PUT /identity/v3/projects/05700a8ee8f942949117a3bf1643b00e/users/5bd3d936dcaf4fd8926e93b17b38114c/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:08:47.416917 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-e1e6e6d4-1761-4774-b26c-504b340ecc05 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:08:47.419086 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-e1e6e6d4-1761-4774-b26c-504b340ecc05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:47.475122 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c70bb4-4692-42ea-9d5f-26b90dffddf5 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:47.475408 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c70bb4-4692-42ea-9d5f-26b90dffddf5 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:47.475680 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c70bb4-4692-42ea-9d5f-26b90dffddf5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:47.487686 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1235/2469] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:08:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 20 09:08:47.497502 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1e6e6d4-1761-4774-b26c-504b340ecc05 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:47.498955 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-e1e6e6d4-1761-4774-b26c-504b340ecc05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:47.508378 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-a8c70bb4-4692-42ea-9d5f-26b90dffddf5 None None] MFA Rules not processed for user `5bd3d936dcaf4fd8926e93b17b38114c`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:08:47.509588 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-a8c70bb4-4692-42ea-9d5f-26b90dffddf5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:47.542048 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=86,keystone:INSERT=4,keystone:DELETE=10 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:47.558394 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1e6e6d4-1761-4774-b26c-504b340ecc05 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:47.562197 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e4ceb38-51e5-4dd7-98fe-8adad6f5923f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:47.562825 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e4ceb38-51e5-4dd7-98fe-8adad6f5923f service neutron] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:47.563100 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e4ceb38-51e5-4dd7-98fe-8adad6f5923f service neutron] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:47.563368 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e4ceb38-51e5-4dd7-98fe-8adad6f5923f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:47.567721 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:47.573954 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e4ceb38-51e5-4dd7-98fe-8adad6f5923f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:47.575554 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e4ceb38-51e5-4dd7-98fe-8adad6f5923f service neutron] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:47.578044 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1235/2470] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:08:47 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 20 09:08:47.582794 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c70bb4-4692-42ea-9d5f-26b90dffddf5 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:47.584247 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-a8c70bb4-4692-42ea-9d5f-26b90dffddf5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:47.585839 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1236/2471] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:08:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:08:47.597911 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c70bb4-4692-42ea-9d5f-26b90dffddf5 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:47.602435 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7a42787-022a-4c47-85be-c456be74fcdf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:47.603057 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7a42787-022a-4c47-85be-c456be74fcdf admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:47.603955 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7a42787-022a-4c47-85be-c456be74fcdf admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:47.604237 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7a42787-022a-4c47-85be-c456be74fcdf admin admin] PATH_INFO: `/v3/projects` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:47.604869 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7a42787-022a-4c47-85be-c456be74fcdf admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:47.607387 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7a42787-022a-4c47-85be-c456be74fcdf admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:47.635198 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1236/2472] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:08:47 2022] POST /identity/v3/projects => generated 381 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 20 09:08:47.639510 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e4ceb38-51e5-4dd7-98fe-8adad6f5923f service neutron] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:47.649505 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e4ceb38-51e5-4dd7-98fe-8adad6f5923f service neutron] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:47.708805 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b98bf7-dbf7-4865-b043-953015ffe9de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:47.709409 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b98bf7-dbf7-4865-b043-953015ffe9de admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:47.709688 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b98bf7-dbf7-4865-b043-953015ffe9de admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:47.710329 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b98bf7-dbf7-4865-b043-953015ffe9de admin admin] PATH_INFO: `/v3/users` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:47.710612 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6b98bf7-dbf7-4865-b043-953015ffe9de admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:47.712535 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6b98bf7-dbf7-4865-b043-953015ffe9de admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:47.740881 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1237/2473] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:08:47 2022] POST /identity/v3/users => generated 291 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:08:47.744955 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7a42787-022a-4c47-85be-c456be74fcdf admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:47.756543 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7a42787-022a-4c47-85be-c456be74fcdf admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:47.761337 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee21acb0-efe0-460e-864d-55d89f90be3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:47.762490 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee21acb0-efe0-460e-864d-55d89f90be3d admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:47.762774 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee21acb0-efe0-460e-864d-55d89f90be3d admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:47.763048 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee21acb0-efe0-460e-864d-55d89f90be3d admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:47.763730 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee21acb0-efe0-460e-864d-55d89f90be3d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:47.765325 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee21acb0-efe0-460e-864d-55d89f90be3d admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:47.781404 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b98bf7-dbf7-4865-b043-953015ffe9de admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:47.782617 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1237/2474] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:08:47 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:08:47.793412 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b98bf7-dbf7-4865-b043-953015ffe9de admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:47.799681 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b17e65e4-821d-427e-aef2-1b11cb45b56d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:47.800364 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b17e65e4-821d-427e-aef2-1b11cb45b56d admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:47.800713 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b17e65e4-821d-427e-aef2-1b11cb45b56d admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:47.801052 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b17e65e4-821d-427e-aef2-1b11cb45b56d admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:47.802214 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b17e65e4-821d-427e-aef2-1b11cb45b56d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:47.804356 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b17e65e4-821d-427e-aef2-1b11cb45b56d admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:47.820826 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1238/2475] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:08:47 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:08:47.821503 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee21acb0-efe0-460e-864d-55d89f90be3d admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:47.833163 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee21acb0-efe0-460e-864d-55d89f90be3d admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:47.839014 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b85a83aa-89eb-4688-b9b0-50d613198df2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:47.842667 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b85a83aa-89eb-4688-b9b0-50d613198df2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:47.842667 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b85a83aa-89eb-4688-b9b0-50d613198df2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:47.842667 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b85a83aa-89eb-4688-b9b0-50d613198df2 admin admin] PATH_INFO: `/v3/projects/1f2e26cf7cf74c4e9c99f42d0740f914/users/4f71a07a9eea473cbfa5aeba3134bf5a/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:47.857354 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b85a83aa-89eb-4688-b9b0-50d613198df2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1f2e26cf7cf74c4e9c99f42d0740f914, user_id=4f71a07a9eea473cbfa5aeba3134bf5a, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:47.860059 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b85a83aa-89eb-4688-b9b0-50d613198df2 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:47.876215 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1238/2476] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:08:47 2022] PUT /identity/v3/projects/1f2e26cf7cf74c4e9c99f42d0740f914/users/4f71a07a9eea473cbfa5aeba3134bf5a/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:08:47.880214 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b17e65e4-821d-427e-aef2-1b11cb45b56d admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:47.882050 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-b17e65e4-821d-427e-aef2-1b11cb45b56d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:47.887246 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77413fec-2dc7-46e3-bcdb-812a28201e9d None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:47.887822 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77413fec-2dc7-46e3-bcdb-812a28201e9d None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:47.888253 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77413fec-2dc7-46e3-bcdb-812a28201e9d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:47.905090 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b17e65e4-821d-427e-aef2-1b11cb45b56d admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:47.926762 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-77413fec-2dc7-46e3-bcdb-812a28201e9d None None] MFA Rules not processed for user `4f71a07a9eea473cbfa5aeba3134bf5a`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:08:47.928013 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-77413fec-2dc7-46e3-bcdb-812a28201e9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:47.982727 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f08a0b80-fcfa-4f35-90bd-4fcacf31a84d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:47.983359 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f08a0b80-fcfa-4f35-90bd-4fcacf31a84d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:47.983634 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f08a0b80-fcfa-4f35-90bd-4fcacf31a84d admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:47.984021 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f08a0b80-fcfa-4f35-90bd-4fcacf31a84d admin admin] PATH_INFO: `/v3/projects/8cd112a3910a4bb49f171cd76c65d9e5` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:47.985140 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f08a0b80-fcfa-4f35-90bd-4fcacf31a84d admin admin] RBAC: Authorizing `identity:delete_project(project_id=8cd112a3910a4bb49f171cd76c65d9e5)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:47.987080 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f08a0b80-fcfa-4f35-90bd-4fcacf31a84d admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:47.993851 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1239/2477] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:08:47 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 20 09:08:47.994793 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77413fec-2dc7-46e3-bcdb-812a28201e9d None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:47.996123 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-77413fec-2dc7-46e3-bcdb-812a28201e9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:48.009126 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77413fec-2dc7-46e3-bcdb-812a28201e9d None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:48.014351 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6bc5d47-b96f-4cf5-8d32-4cb03b65566c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:48.015011 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6bc5d47-b96f-4cf5-8d32-4cb03b65566c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:48.015289 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6bc5d47-b96f-4cf5-8d32-4cb03b65566c admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:48.015552 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6bc5d47-b96f-4cf5-8d32-4cb03b65566c admin admin] PATH_INFO: `/v3/users/fd615f9fd25c4602a4609a1444f725fa` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:48.016990 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6bc5d47-b96f-4cf5-8d32-4cb03b65566c admin admin] RBAC: Authorizing `identity:delete_user(user_id=fd615f9fd25c4602a4609a1444f725fa)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:48.019048 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6bc5d47-b96f-4cf5-8d32-4cb03b65566c admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:48.035531 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-f08a0b80-fcfa-4f35-90bd-4fcacf31a84d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8cd112a3910a4bb49f171cd76c65d9e5', 'request_id': 'req-f08a0b80-fcfa-4f35-90bd-4fcacf31a84d'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:48.042092 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-f08a0b80-fcfa-4f35-90bd-4fcacf31a84d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8cd112a3910a4bb49f171cd76c65d9e5', 'request_id': 'req-f08a0b80-fcfa-4f35-90bd-4fcacf31a84d'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:48.049289 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-f08a0b80-fcfa-4f35-90bd-4fcacf31a84d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8cd112a3910a4bb49f171cd76c65d9e5', 'request_id': 'req-f08a0b80-fcfa-4f35-90bd-4fcacf31a84d'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:48.051767 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=91,keystone:INSERT=3,keystone:DELETE=6 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:48.065004 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-f08a0b80-fcfa-4f35-90bd-4fcacf31a84d admin admin] The token cache is being invalidate because project 8cd112a3910a4bb49f171cd76c65d9e5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81561) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:08:48.065296 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-f08a0b80-fcfa-4f35-90bd-4fcacf31a84d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:48.065961 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-f08a0b80-fcfa-4f35-90bd-4fcacf31a84d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:48.067520 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1239/2478] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:08:47 2022] DELETE /identity/v3/projects/8cd112a3910a4bb49f171cd76c65d9e5 => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:08:48.068260 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f08a0b80-fcfa-4f35-90bd-4fcacf31a84d admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:48.069557 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-f08a0b80-fcfa-4f35-90bd-4fcacf31a84d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:48.071550 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-a6bc5d47-b96f-4cf5-8d32-4cb03b65566c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fd615f9fd25c4602a4609a1444f725fa'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:48.080016 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-a6bc5d47-b96f-4cf5-8d32-4cb03b65566c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fd615f9fd25c4602a4609a1444f725fa'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:48.093932 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-a6bc5d47-b96f-4cf5-8d32-4cb03b65566c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fd615f9fd25c4602a4609a1444f725fa'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:48.100655 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-a6bc5d47-b96f-4cf5-8d32-4cb03b65566c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fd615f9fd25c4602a4609a1444f725fa'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:48.101329 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-a6bc5d47-b96f-4cf5-8d32-4cb03b65566c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fd615f9fd25c4602a4609a1444f725fa'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:48.103431 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=87,keystone:INSERT=3,keystone:DELETE=10 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:48.108211 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1240/2479] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:08:47 2022] DELETE /identity/v3/users/fd615f9fd25c4602a4609a1444f725fa => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:08:48.113181 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27a412cc-af90-498d-8b61-d241f2b07671 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:48.113991 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27a412cc-af90-498d-8b61-d241f2b07671 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:48.114273 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27a412cc-af90-498d-8b61-d241f2b07671 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:48.134352 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-27a412cc-af90-498d-8b61-d241f2b07671 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:08:48.135576 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-27a412cc-af90-498d-8b61-d241f2b07671 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:48.135772 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f08a0b80-fcfa-4f35-90bd-4fcacf31a84d admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:48.193556 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a042024-e1c8-4894-bfe6-85a9ccc3ca75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:48.194164 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a042024-e1c8-4894-bfe6-85a9ccc3ca75 admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:48.194443 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a042024-e1c8-4894-bfe6-85a9ccc3ca75 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:48.194746 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a042024-e1c8-4894-bfe6-85a9ccc3ca75 admin admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:48.195299 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1241/2480] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:08:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:08:48.195404 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a042024-e1c8-4894-bfe6-85a9ccc3ca75 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:48.196550 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adeb36c7-38eb-4abd-8f74-1189969280bf None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:48.196863 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adeb36c7-38eb-4abd-8f74-1189969280bf None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:48.197165 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adeb36c7-38eb-4abd-8f74-1189969280bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:48.197265 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a042024-e1c8-4894-bfe6-85a9ccc3ca75 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:48.218946 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-adeb36c7-38eb-4abd-8f74-1189969280bf None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:08:48.220153 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-adeb36c7-38eb-4abd-8f74-1189969280bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:48.230099 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1240/2481] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:08:48 2022] POST /identity/v3/projects => generated 383 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:08:48.230276 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1242/2482] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:08:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:08:48.230404 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0519ff48-db34-42b7-9032-532d57d67239 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:48.230404 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0519ff48-db34-42b7-9032-532d57d67239 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:48.230404 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0519ff48-db34-42b7-9032-532d57d67239 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:48.232531 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adeb36c7-38eb-4abd-8f74-1189969280bf None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:48.234160 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-adeb36c7-38eb-4abd-8f74-1189969280bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:48.246974 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adeb36c7-38eb-4abd-8f74-1189969280bf None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:48.248998 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-0519ff48-db34-42b7-9032-532d57d67239 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:08:48.250240 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-0519ff48-db34-42b7-9032-532d57d67239 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:48.302337 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15d112f7-3f8f-4a41-85a5-25e65b71b393 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:48.303111 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d112f7-3f8f-4a41-85a5-25e65b71b393 admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:48.303391 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d112f7-3f8f-4a41-85a5-25e65b71b393 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:48.303660 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d112f7-3f8f-4a41-85a5-25e65b71b393 admin admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:48.304311 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15d112f7-3f8f-4a41-85a5-25e65b71b393 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:48.306247 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15d112f7-3f8f-4a41-85a5-25e65b71b393 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:48.311022 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1241/2483] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:08:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:08:48.311856 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0519ff48-db34-42b7-9032-532d57d67239 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:48.313111 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-0519ff48-db34-42b7-9032-532d57d67239 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:48.332058 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1243/2484] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:08:48 2022] POST /identity/v3/users => generated 298 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 20 09:08:48.332058 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15d112f7-3f8f-4a41-85a5-25e65b71b393 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:48.339261 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15d112f7-3f8f-4a41-85a5-25e65b71b393 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:48.342778 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:INSERT=2,keystone:DELETE=1,keystone:SELECT=97 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:48.345432 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-510762f8-a945-458d-8bcf-e18bed8db6fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:48.346009 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-510762f8-a945-458d-8bcf-e18bed8db6fc admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:48.346280 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-510762f8-a945-458d-8bcf-e18bed8db6fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:48.346586 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-510762f8-a945-458d-8bcf-e18bed8db6fc admin admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:48.347221 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-510762f8-a945-458d-8bcf-e18bed8db6fc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 09:08:48.347519 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-510762f8-a945-458d-8bcf-e18bed8db6fc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:48.349404 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-510762f8-a945-458d-8bcf-e18bed8db6fc admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:48.362914 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1244/2485] 10.222.0.36 () {58 vars in 1153 bytes} [Thu Oct 20 09:08:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 09:08:48.364077 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-510762f8-a945-458d-8bcf-e18bed8db6fc admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:48.375457 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-510762f8-a945-458d-8bcf-e18bed8db6fc admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:48.380838 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59471d71-a3b8-4f41-b57d-33153bc51f12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:48.381440 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59471d71-a3b8-4f41-b57d-33153bc51f12 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:48.381720 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59471d71-a3b8-4f41-b57d-33153bc51f12 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:48.381992 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59471d71-a3b8-4f41-b57d-33153bc51f12 admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:48.382660 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59471d71-a3b8-4f41-b57d-33153bc51f12 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:48.384292 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59471d71-a3b8-4f41-b57d-33153bc51f12 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:48.384896 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0519ff48-db34-42b7-9032-532d57d67239 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:48.389539 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31e4cd46-16dc-43d0-b50b-54fefbcaa59d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:48.390139 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31e4cd46-16dc-43d0-b50b-54fefbcaa59d service neutron] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:48.390437 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31e4cd46-16dc-43d0-b50b-54fefbcaa59d service neutron] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:48.390731 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31e4cd46-16dc-43d0-b50b-54fefbcaa59d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:48.395173 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1245/2486] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:08:48 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:08:48.396024 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59471d71-a3b8-4f41-b57d-33153bc51f12 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:48.400632 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31e4cd46-16dc-43d0-b50b-54fefbcaa59d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:48.402330 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31e4cd46-16dc-43d0-b50b-54fefbcaa59d service neutron] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:48.406484 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59471d71-a3b8-4f41-b57d-33153bc51f12 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:48.411209 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c1954e1-892e-46d4-bffe-26844fd416d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:48.411772 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c1954e1-892e-46d4-bffe-26844fd416d4 admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:48.412041 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c1954e1-892e-46d4-bffe-26844fd416d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:48.412304 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c1954e1-892e-46d4-bffe-26844fd416d4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:48.412912 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c1954e1-892e-46d4-bffe-26844fd416d4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:48.413080 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1242/2487] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:08:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:08:48.413704 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31e4cd46-16dc-43d0-b50b-54fefbcaa59d service neutron] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:48.414810 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c1954e1-892e-46d4-bffe-26844fd416d4 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:48.423316 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31e4cd46-16dc-43d0-b50b-54fefbcaa59d service neutron] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:48.429298 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90e95e6c-5a06-4d62-bc39-300aeb4410f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:48.430059 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e95e6c-5a06-4d62-bc39-300aeb4410f0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:48.430346 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e95e6c-5a06-4d62-bc39-300aeb4410f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:48.430809 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e95e6c-5a06-4d62-bc39-300aeb4410f0 admin admin] PATH_INFO: `/v3/projects/f6b5cfda167247ef89d9e4139967492b/users/6d5202a0f4db4cb28d7505120ee8715a/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:48.447415 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1246/2488] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:08:48 2022] POST /identity/v3/projects => generated 389 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:08:48.448447 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c1954e1-892e-46d4-bffe-26844fd416d4 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:48.453333 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90e95e6c-5a06-4d62-bc39-300aeb4410f0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f6b5cfda167247ef89d9e4139967492b, user_id=6d5202a0f4db4cb28d7505120ee8715a, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:48.456373 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90e95e6c-5a06-4d62-bc39-300aeb4410f0 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:48.474455 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1243/2489] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:08:48 2022] PUT /identity/v3/projects/f6b5cfda167247ef89d9e4139967492b/users/6d5202a0f4db4cb28d7505120ee8715a/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:08:48.478185 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90e95e6c-5a06-4d62-bc39-300aeb4410f0 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:48.503430 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:48.526229 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c1954e1-892e-46d4-bffe-26844fd416d4 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:48.554784 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90e95e6c-5a06-4d62-bc39-300aeb4410f0 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:48.560729 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb673d75-3118-4076-8663-2015c3a696a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:48.561563 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb673d75-3118-4076-8663-2015c3a696a5 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:48.561844 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb673d75-3118-4076-8663-2015c3a696a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:48.565021 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb673d75-3118-4076-8663-2015c3a696a5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:48.565021 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb673d75-3118-4076-8663-2015c3a696a5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:48.565021 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb673d75-3118-4076-8663-2015c3a696a5 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:48.577139 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1244/2490] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:08:48 2022] GET /identity/v3/roles => generated 1420 bytes in 99 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 20 09:08:48.581633 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb673d75-3118-4076-8663-2015c3a696a5 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:48.593288 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb673d75-3118-4076-8663-2015c3a696a5 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:48.593566 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fd831e3-7dba-45ce-909a-d7b4003f7d5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:48.594179 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd831e3-7dba-45ce-909a-d7b4003f7d5e admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:48.594460 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd831e3-7dba-45ce-909a-d7b4003f7d5e admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:48.594765 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd831e3-7dba-45ce-909a-d7b4003f7d5e admin admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:48.595393 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fd831e3-7dba-45ce-909a-d7b4003f7d5e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:48.597579 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fd831e3-7dba-45ce-909a-d7b4003f7d5e admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:48.598600 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6bfcf06-6770-4d93-8c1a-15aa7696dda1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:48.599378 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6bfcf06-6770-4d93-8c1a-15aa7696dda1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:48.599761 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6bfcf06-6770-4d93-8c1a-15aa7696dda1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:48.600161 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6bfcf06-6770-4d93-8c1a-15aa7696dda1 admin admin] PATH_INFO: `/v3/projects/f6b5cfda167247ef89d9e4139967492b/users/6d5202a0f4db4cb28d7505120ee8715a/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:48.602232 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6bfcf06-6770-4d93-8c1a-15aa7696dda1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f6b5cfda167247ef89d9e4139967492b, user_id=6d5202a0f4db4cb28d7505120ee8715a, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:48.605014 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6bfcf06-6770-4d93-8c1a-15aa7696dda1 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:48.618131 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1245/2491] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:08:48 2022] PUT /identity/v3/projects/f6b5cfda167247ef89d9e4139967492b/users/6d5202a0f4db4cb28d7505120ee8715a/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:08:48.622137 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1247/2492] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:08:48 2022] POST /identity/v3/users => generated 295 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 20 09:08:48.626105 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6bfcf06-6770-4d93-8c1a-15aa7696dda1 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:48.629322 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a94c959-d2fd-4376-8251-36323a65207e None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:48.629730 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a94c959-d2fd-4376-8251-36323a65207e None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:48.630004 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a94c959-d2fd-4376-8251-36323a65207e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:48.663615 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-1a94c959-d2fd-4376-8251-36323a65207e None None] MFA Rules not processed for user `6d5202a0f4db4cb28d7505120ee8715a`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:08:48.664925 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-1a94c959-d2fd-4376-8251-36323a65207e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:48.698743 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6bfcf06-6770-4d93-8c1a-15aa7696dda1 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:48.707211 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e175b9b-2514-4a25-8036-bbb5ce55451e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:48.708076 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e175b9b-2514-4a25-8036-bbb5ce55451e admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:48.708358 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e175b9b-2514-4a25-8036-bbb5ce55451e admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:48.708629 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e175b9b-2514-4a25-8036-bbb5ce55451e admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:48.709298 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e175b9b-2514-4a25-8036-bbb5ce55451e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:48.711063 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e175b9b-2514-4a25-8036-bbb5ce55451e admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:48.724542 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1246/2493] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:08:48 2022] GET /identity/v3/roles => generated 1420 bytes in 99 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:08:48.728988 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e175b9b-2514-4a25-8036-bbb5ce55451e admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:48.731494 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-0e175b9b-2514-4a25-8036-bbb5ce55451e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:48.744944 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e175b9b-2514-4a25-8036-bbb5ce55451e admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:48.747811 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1248/2494] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:08:48 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:08:48.748542 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a94c959-d2fd-4376-8251-36323a65207e None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:48.751382 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d81eb642-d75a-4c3d-a93c-3451f96d218c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:48.758255 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d81eb642-d75a-4c3d-a93c-3451f96d218c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:48.758581 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d81eb642-d75a-4c3d-a93c-3451f96d218c admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:48.758715 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a94c959-d2fd-4376-8251-36323a65207e None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:48.758867 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d81eb642-d75a-4c3d-a93c-3451f96d218c admin admin] PATH_INFO: `/v3/projects/73148d6d2be14453b3614b4276672ca1` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:48.760051 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d81eb642-d75a-4c3d-a93c-3451f96d218c admin admin] RBAC: Authorizing `identity:delete_project(project_id=73148d6d2be14453b3614b4276672ca1)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:48.762018 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d81eb642-d75a-4c3d-a93c-3451f96d218c admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:48.763430 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a11bbc02-f55e-4327-a3e4-bfe8e2625f94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:48.764029 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a11bbc02-f55e-4327-a3e4-bfe8e2625f94 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:48.764292 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a11bbc02-f55e-4327-a3e4-bfe8e2625f94 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:48.764565 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a11bbc02-f55e-4327-a3e4-bfe8e2625f94 admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:48.765193 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a11bbc02-f55e-4327-a3e4-bfe8e2625f94 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:48.766804 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a11bbc02-f55e-4327-a3e4-bfe8e2625f94 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:48.776817 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1249/2495] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:08:48 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:08:48.777642 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a11bbc02-f55e-4327-a3e4-bfe8e2625f94 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:48.779289 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-a11bbc02-f55e-4327-a3e4-bfe8e2625f94 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:48.782012 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:48.814218 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-d81eb642-d75a-4c3d-a93c-3451f96d218c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '73148d6d2be14453b3614b4276672ca1', 'request_id': 'req-d81eb642-d75a-4c3d-a93c-3451f96d218c'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:48.820627 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-d81eb642-d75a-4c3d-a93c-3451f96d218c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '73148d6d2be14453b3614b4276672ca1', 'request_id': 'req-d81eb642-d75a-4c3d-a93c-3451f96d218c'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:48.826735 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-d81eb642-d75a-4c3d-a93c-3451f96d218c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '73148d6d2be14453b3614b4276672ca1', 'request_id': 'req-d81eb642-d75a-4c3d-a93c-3451f96d218c'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:48.835981 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:48.840541 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-d81eb642-d75a-4c3d-a93c-3451f96d218c admin admin] The token cache is being invalidate because project 73148d6d2be14453b3614b4276672ca1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81561) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:08:48.840913 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-d81eb642-d75a-4c3d-a93c-3451f96d218c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:48.841584 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-d81eb642-d75a-4c3d-a93c-3451f96d218c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:48.843094 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1247/2496] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:08:48 2022] DELETE /identity/v3/projects/73148d6d2be14453b3614b4276672ca1 => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:08:48.843294 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a11bbc02-f55e-4327-a3e4-bfe8e2625f94 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:48.843820 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d81eb642-d75a-4c3d-a93c-3451f96d218c admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:48.845122 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-d81eb642-d75a-4c3d-a93c-3451f96d218c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:48.896504 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fa6686c-28b6-40c6-ba64-af28da4d1feb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:48.897107 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa6686c-28b6-40c6-ba64-af28da4d1feb service cinder] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:48.897520 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa6686c-28b6-40c6-ba64-af28da4d1feb service cinder] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:48.897798 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa6686c-28b6-40c6-ba64-af28da4d1feb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:48.899595 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-7fa6686c-28b6-40c6-ba64-af28da4d1feb service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:48.910324 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d81eb642-d75a-4c3d-a93c-3451f96d218c admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:48.914962 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00cd337e-c2ab-4e8c-96fd-cb5eaf03b200 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:48.915609 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00cd337e-c2ab-4e8c-96fd-cb5eaf03b200 admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:48.915882 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00cd337e-c2ab-4e8c-96fd-cb5eaf03b200 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:48.916244 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00cd337e-c2ab-4e8c-96fd-cb5eaf03b200 admin admin] PATH_INFO: `/v3/projects/ae904361755c44d9842b1ce7829bb4ce/users/aaef5a9a56d64c1badace8afee33f438/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:48.932102 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00cd337e-c2ab-4e8c-96fd-cb5eaf03b200 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ae904361755c44d9842b1ce7829bb4ce, user_id=aaef5a9a56d64c1badace8afee33f438, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:48.934807 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00cd337e-c2ab-4e8c-96fd-cb5eaf03b200 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:48.947757 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1248/2497] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:08:48 2022] PUT /identity/v3/projects/ae904361755c44d9842b1ce7829bb4ce/users/aaef5a9a56d64c1badace8afee33f438/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:08:48.948561 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00cd337e-c2ab-4e8c-96fd-cb5eaf03b200 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:48.950233 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-00cd337e-c2ab-4e8c-96fd-cb5eaf03b200 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:48.961262 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fa6686c-28b6-40c6-ba64-af28da4d1feb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:48.962810 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fa6686c-28b6-40c6-ba64-af28da4d1feb service cinder] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:48.972271 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1250/2498] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:08:48 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 195 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:08:48.973534 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1bd1f1-b20c-4afd-b986-36c7803d432c None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:48.973891 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1bd1f1-b20c-4afd-b986-36c7803d432c None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:48.974230 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1bd1f1-b20c-4afd-b986-36c7803d432c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:49.006102 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-7c1bd1f1-b20c-4afd-b986-36c7803d432c None None] MFA Rules not processed for user `aaef5a9a56d64c1badace8afee33f438`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:08:49.007559 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-7c1bd1f1-b20c-4afd-b986-36c7803d432c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:49.017109 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00cd337e-c2ab-4e8c-96fd-cb5eaf03b200 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:49.022639 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbacb4c2-9996-4879-9215-edebf01011a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:49.023267 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbacb4c2-9996-4879-9215-edebf01011a5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:49.023538 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbacb4c2-9996-4879-9215-edebf01011a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:49.023811 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbacb4c2-9996-4879-9215-edebf01011a5 admin admin] PATH_INFO: `/v3/users/bed4c297f1074bd8b71c906f8d17f79f` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:49.024943 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbacb4c2-9996-4879-9215-edebf01011a5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bed4c297f1074bd8b71c906f8d17f79f)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:49.026883 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbacb4c2-9996-4879-9215-edebf01011a5 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:49.064067 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:49.080354 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-bbacb4c2-9996-4879-9215-edebf01011a5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bed4c297f1074bd8b71c906f8d17f79f'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:49.085355 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1251/2499] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:08:48 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:08:49.091013 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-bbacb4c2-9996-4879-9215-edebf01011a5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bed4c297f1074bd8b71c906f8d17f79f'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:49.098380 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c1bd1f1-b20c-4afd-b986-36c7803d432c None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:49.100279 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-7c1bd1f1-b20c-4afd-b986-36c7803d432c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:49.109563 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-bbacb4c2-9996-4879-9215-edebf01011a5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bed4c297f1074bd8b71c906f8d17f79f'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:49.120372 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-bbacb4c2-9996-4879-9215-edebf01011a5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bed4c297f1074bd8b71c906f8d17f79f'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:49.121455 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-bbacb4c2-9996-4879-9215-edebf01011a5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bed4c297f1074bd8b71c906f8d17f79f'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:49.129637 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1249/2500] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:08:48 2022] DELETE /identity/v3/users/bed4c297f1074bd8b71c906f8d17f79f => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:08:49.162176 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:49.166930 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c1bd1f1-b20c-4afd-b986-36c7803d432c None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:49.223750 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5866ad18-0a44-4b5b-b3e9-954f0f884be5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:49.224652 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5866ad18-0a44-4b5b-b3e9-954f0f884be5 service cinder] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:49.225087 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5866ad18-0a44-4b5b-b3e9-954f0f884be5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:49.225544 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5866ad18-0a44-4b5b-b3e9-954f0f884be5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:49.227784 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-5866ad18-0a44-4b5b-b3e9-954f0f884be5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:49.307078 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5866ad18-0a44-4b5b-b3e9-954f0f884be5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:49.308818 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5866ad18-0a44-4b5b-b3e9-954f0f884be5 service cinder] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:49.321054 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1252/2501] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:08:49 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 223 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 20 09:08:49.473336 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbacb4c2-9996-4879-9215-edebf01011a5 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:49.475314 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-bbacb4c2-9996-4879-9215-edebf01011a5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:49.552328 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbacb4c2-9996-4879-9215-edebf01011a5 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:49.557453 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1552a71d-4028-4482-aad4-aa2124823ff8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:49.558173 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1552a71d-4028-4482-aad4-aa2124823ff8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:49.559055 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1552a71d-4028-4482-aad4-aa2124823ff8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:49.559383 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1552a71d-4028-4482-aad4-aa2124823ff8 admin admin] PATH_INFO: `/v3/projects/de1e4f0f8d59429db905008319c10a31` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:49.560566 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1552a71d-4028-4482-aad4-aa2124823ff8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=de1e4f0f8d59429db905008319c10a31)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:49.562763 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1552a71d-4028-4482-aad4-aa2124823ff8 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:49.615066 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-1552a71d-4028-4482-aad4-aa2124823ff8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'de1e4f0f8d59429db905008319c10a31', 'request_id': 'req-1552a71d-4028-4482-aad4-aa2124823ff8'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:49.622676 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-1552a71d-4028-4482-aad4-aa2124823ff8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'de1e4f0f8d59429db905008319c10a31', 'request_id': 'req-1552a71d-4028-4482-aad4-aa2124823ff8'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:49.631669 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-1552a71d-4028-4482-aad4-aa2124823ff8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'de1e4f0f8d59429db905008319c10a31', 'request_id': 'req-1552a71d-4028-4482-aad4-aa2124823ff8'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:49.655079 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-1552a71d-4028-4482-aad4-aa2124823ff8 admin admin] The token cache is being invalidate because project de1e4f0f8d59429db905008319c10a31 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81561) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:08:49.655378 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-1552a71d-4028-4482-aad4-aa2124823ff8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:49.656100 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-1552a71d-4028-4482-aad4-aa2124823ff8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:49.658386 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1250/2502] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:08:49 2022] DELETE /identity/v3/projects/de1e4f0f8d59429db905008319c10a31 => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:08:49.663454 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f98ec92f-2666-47fd-a2d0-756db564771b None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:49.663967 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f98ec92f-2666-47fd-a2d0-756db564771b None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:49.664089 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f98ec92f-2666-47fd-a2d0-756db564771b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:49.685396 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-f98ec92f-2666-47fd-a2d0-756db564771b None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:08:49.686721 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-f98ec92f-2666-47fd-a2d0-756db564771b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:49.745934 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1253/2503] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:08:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:08:49.753978 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe1a7276-c571-4e83-919f-d377b05e000e None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:49.754267 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe1a7276-c571-4e83-919f-d377b05e000e None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:49.754571 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe1a7276-c571-4e83-919f-d377b05e000e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:49.773888 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-fe1a7276-c571-4e83-919f-d377b05e000e None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:08:49.775140 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-fe1a7276-c571-4e83-919f-d377b05e000e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:49.783498 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1251/2504] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:08:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:08:49.787730 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f98ec92f-2666-47fd-a2d0-756db564771b None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:49.797790 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f98ec92f-2666-47fd-a2d0-756db564771b None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:49.802370 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4b8b7bf-1433-4491-8226-5cfef5213247 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:49.803000 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4b8b7bf-1433-4491-8226-5cfef5213247 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:49.803275 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4b8b7bf-1433-4491-8226-5cfef5213247 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:49.803550 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4b8b7bf-1433-4491-8226-5cfef5213247 admin admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:49.804190 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4b8b7bf-1433-4491-8226-5cfef5213247 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 09:08:49.804487 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4b8b7bf-1433-4491-8226-5cfef5213247 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:49.806067 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4b8b7bf-1433-4491-8226-5cfef5213247 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:49.817634 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1254/2505] 10.222.0.36 () {58 vars in 1153 bytes} [Thu Oct 20 09:08:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 09:08:49.821893 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe1a7276-c571-4e83-919f-d377b05e000e None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:49.831376 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe1a7276-c571-4e83-919f-d377b05e000e None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:49.836032 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bbe8120-e818-476d-ad28-a7d0b48ee423 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:49.836621 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bbe8120-e818-476d-ad28-a7d0b48ee423 admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:49.836903 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bbe8120-e818-476d-ad28-a7d0b48ee423 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:49.837173 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bbe8120-e818-476d-ad28-a7d0b48ee423 admin admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:49.837843 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bbe8120-e818-476d-ad28-a7d0b48ee423 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:49.840335 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bbe8120-e818-476d-ad28-a7d0b48ee423 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:49.866915 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1252/2506] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:08:49 2022] POST /identity/v3/projects => generated 375 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:08:49.871069 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4b8b7bf-1433-4491-8226-5cfef5213247 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:49.881012 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4b8b7bf-1433-4491-8226-5cfef5213247 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:49.923683 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:49.936819 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7dc640c-c6c4-4f48-97a8-248dab0afa90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:49.937425 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7dc640c-c6c4-4f48-97a8-248dab0afa90 admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:49.937702 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7dc640c-c6c4-4f48-97a8-248dab0afa90 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:49.937970 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7dc640c-c6c4-4f48-97a8-248dab0afa90 admin admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:49.938629 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7dc640c-c6c4-4f48-97a8-248dab0afa90 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:49.940534 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7dc640c-c6c4-4f48-97a8-248dab0afa90 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:49.960376 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1255/2507] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:08:49 2022] POST /identity/v3/users => generated 288 bytes in 90 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 20 09:08:49.964450 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bbe8120-e818-476d-ad28-a7d0b48ee423 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:49.975300 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bbe8120-e818-476d-ad28-a7d0b48ee423 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:49.979783 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c37593f3-8a4b-4005-9581-7f73b7de1aba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:49.980969 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37593f3-8a4b-4005-9581-7f73b7de1aba admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:49.981245 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37593f3-8a4b-4005-9581-7f73b7de1aba admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:49.981514 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37593f3-8a4b-4005-9581-7f73b7de1aba admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:49.982144 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c37593f3-8a4b-4005-9581-7f73b7de1aba admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:49.983798 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c37593f3-8a4b-4005-9581-7f73b7de1aba admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:49.985892 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=85,keystone:DELETE=12,keystone:INSERT=3 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:49.995620 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1253/2508] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:08:49 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 20 09:08:49.999416 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7dc640c-c6c4-4f48-97a8-248dab0afa90 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:50.009636 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7dc640c-c6c4-4f48-97a8-248dab0afa90 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:50.014830 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c3903f-50b7-4905-87c7-028e4bc8c6d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:50.015407 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c3903f-50b7-4905-87c7-028e4bc8c6d7 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:50.015685 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c3903f-50b7-4905-87c7-028e4bc8c6d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:50.015956 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c3903f-50b7-4905-87c7-028e4bc8c6d7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:50.016593 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61c3903f-50b7-4905-87c7-028e4bc8c6d7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:50.018184 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61c3903f-50b7-4905-87c7-028e4bc8c6d7 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:50.029319 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1256/2509] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:08:49 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:08:50.033564 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c37593f3-8a4b-4005-9581-7f73b7de1aba admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:50.043437 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c37593f3-8a4b-4005-9581-7f73b7de1aba admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:50.047950 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18c4abf5-1b4a-477c-943c-6d63bf7a6fd4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:50.048590 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c4abf5-1b4a-477c-943c-6d63bf7a6fd4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:50.048866 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c4abf5-1b4a-477c-943c-6d63bf7a6fd4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:50.049141 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c4abf5-1b4a-477c-943c-6d63bf7a6fd4 admin admin] PATH_INFO: `/v3/projects/6ea45b923e70427ea00dc803fc8118ed/users/1b7f770792fb46dcb31954ad136ea64a/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:50.063357 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18c4abf5-1b4a-477c-943c-6d63bf7a6fd4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6ea45b923e70427ea00dc803fc8118ed, user_id=1b7f770792fb46dcb31954ad136ea64a, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:50.066000 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18c4abf5-1b4a-477c-943c-6d63bf7a6fd4 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:50.078829 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1254/2510] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:08:50 2022] PUT /identity/v3/projects/6ea45b923e70427ea00dc803fc8118ed/users/1b7f770792fb46dcb31954ad136ea64a/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:08:50.090647 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39e8f303-5695-435e-8058-0283b24afb49 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:50.090946 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39e8f303-5695-435e-8058-0283b24afb49 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:50.091262 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39e8f303-5695-435e-8058-0283b24afb49 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:50.119309 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-39e8f303-5695-435e-8058-0283b24afb49 None None] MFA Rules not processed for user `1b7f770792fb46dcb31954ad136ea64a`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:08:50.120506 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-39e8f303-5695-435e-8058-0283b24afb49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:50.184908 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1257/2511] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:08:50 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:08:50.189988 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18c4abf5-1b4a-477c-943c-6d63bf7a6fd4 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:50.251228 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18c4abf5-1b4a-477c-943c-6d63bf7a6fd4 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:50.255894 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80cc83ee-64c4-465a-b84e-aa0e55018be4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:50.256482 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80cc83ee-64c4-465a-b84e-aa0e55018be4 admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:50.256757 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80cc83ee-64c4-465a-b84e-aa0e55018be4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:50.257024 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80cc83ee-64c4-465a-b84e-aa0e55018be4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:50.257636 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80cc83ee-64c4-465a-b84e-aa0e55018be4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:50.259570 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80cc83ee-64c4-465a-b84e-aa0e55018be4 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:50.286301 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1255/2512] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:08:50 2022] POST /identity/v3/projects => generated 375 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:08:50.290262 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39e8f303-5695-435e-8058-0283b24afb49 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:50.300955 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39e8f303-5695-435e-8058-0283b24afb49 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:50.359796 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d338aacd-91f3-4678-b486-ab99eade856a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:50.360410 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d338aacd-91f3-4678-b486-ab99eade856a admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:50.360716 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d338aacd-91f3-4678-b486-ab99eade856a admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:50.360987 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d338aacd-91f3-4678-b486-ab99eade856a admin admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:50.361737 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d338aacd-91f3-4678-b486-ab99eade856a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:50.364401 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d338aacd-91f3-4678-b486-ab99eade856a admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:50.390288 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1258/2513] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:08:50 2022] POST /identity/v3/users => generated 294 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:08:50.392543 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80cc83ee-64c4-465a-b84e-aa0e55018be4 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:50.403451 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80cc83ee-64c4-465a-b84e-aa0e55018be4 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:50.413069 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb817f7b-f3ff-4c86-a954-a05ec91d8978 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:50.413069 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb817f7b-f3ff-4c86-a954-a05ec91d8978 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:50.413069 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb817f7b-f3ff-4c86-a954-a05ec91d8978 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:50.413069 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb817f7b-f3ff-4c86-a954-a05ec91d8978 admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:50.413488 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb817f7b-f3ff-4c86-a954-a05ec91d8978 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:50.413488 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb817f7b-f3ff-4c86-a954-a05ec91d8978 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:50.425164 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1256/2514] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:08:50 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:08:50.429483 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d338aacd-91f3-4678-b486-ab99eade856a admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:50.440701 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d338aacd-91f3-4678-b486-ab99eade856a admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:50.445601 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01804926-59df-4400-aa4f-a9a75b5a60af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:50.446243 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01804926-59df-4400-aa4f-a9a75b5a60af admin admin] REQUEST_METHOD: `PUT` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:50.446564 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01804926-59df-4400-aa4f-a9a75b5a60af admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:50.446842 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01804926-59df-4400-aa4f-a9a75b5a60af admin admin] PATH_INFO: `/v3/projects/efeaf19187264177a120e64ea7d15e05/users/5a076dbe4d7e42d99c5c895885517fa4/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:50.463757 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01804926-59df-4400-aa4f-a9a75b5a60af admin admin] RBAC: Authorizing `identity:create_grant(project_id=efeaf19187264177a120e64ea7d15e05, user_id=5a076dbe4d7e42d99c5c895885517fa4, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:50.466861 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01804926-59df-4400-aa4f-a9a75b5a60af admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:50.480738 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1259/2515] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:08:50 2022] PUT /identity/v3/projects/efeaf19187264177a120e64ea7d15e05/users/5a076dbe4d7e42d99c5c895885517fa4/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:08:50.484566 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb817f7b-f3ff-4c86-a954-a05ec91d8978 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:50.501012 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb817f7b-f3ff-4c86-a954-a05ec91d8978 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:50.533189 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01804926-59df-4400-aa4f-a9a75b5a60af admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:50.535288 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-01804926-59df-4400-aa4f-a9a75b5a60af admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:50.560707 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:50.573319 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4338925b-48a3-4a53-8b6b-0cc8d57c9add admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:50.573989 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4338925b-48a3-4a53-8b6b-0cc8d57c9add admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:50.574964 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4338925b-48a3-4a53-8b6b-0cc8d57c9add admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:50.575421 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4338925b-48a3-4a53-8b6b-0cc8d57c9add admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:50.576017 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4338925b-48a3-4a53-8b6b-0cc8d57c9add admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:50.577711 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4338925b-48a3-4a53-8b6b-0cc8d57c9add admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:50.590523 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1257/2516] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:08:50 2022] GET /identity/v3/roles => generated 1420 bytes in 106 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:08:50.594190 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4338925b-48a3-4a53-8b6b-0cc8d57c9add admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:50.604063 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4338925b-48a3-4a53-8b6b-0cc8d57c9add admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:50.604375 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01804926-59df-4400-aa4f-a9a75b5a60af admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:50.608273 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6eef3dc-e7cc-4173-b028-807d8b7cd6d5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:50.609026 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6eef3dc-e7cc-4173-b028-807d8b7cd6d5 service cinder] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:50.609147 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e8be97-6203-480f-ba5b-152713d73dbf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:50.609504 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6eef3dc-e7cc-4173-b028-807d8b7cd6d5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:50.609684 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e8be97-6203-480f-ba5b-152713d73dbf admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:50.609961 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6eef3dc-e7cc-4173-b028-807d8b7cd6d5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:50.610090 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e8be97-6203-480f-ba5b-152713d73dbf admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:50.610221 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e8be97-6203-480f-ba5b-152713d73dbf admin admin] PATH_INFO: `/v3/projects/efeaf19187264177a120e64ea7d15e05/users/5a076dbe4d7e42d99c5c895885517fa4/roles/d9d3bc62f6e8453488c74856ef333bb3` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:50.612134 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44e8be97-6203-480f-ba5b-152713d73dbf admin admin] RBAC: Authorizing `identity:create_grant(project_id=efeaf19187264177a120e64ea7d15e05, user_id=5a076dbe4d7e42d99c5c895885517fa4, role_id=d9d3bc62f6e8453488c74856ef333bb3)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:50.612320 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-d6eef3dc-e7cc-4173-b028-807d8b7cd6d5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:50.614826 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44e8be97-6203-480f-ba5b-152713d73dbf admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:50.627111 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1258/2517] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:08:50 2022] PUT /identity/v3/projects/efeaf19187264177a120e64ea7d15e05/users/5a076dbe4d7e42d99c5c895885517fa4/roles/d9d3bc62f6e8453488c74856ef333bb3 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:08:50.637827 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb475812-9d5d-4488-9397-6ac8a41c905e None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:50.638111 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb475812-9d5d-4488-9397-6ac8a41c905e None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:50.638383 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb475812-9d5d-4488-9397-6ac8a41c905e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:50.665706 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-cb475812-9d5d-4488-9397-6ac8a41c905e None None] MFA Rules not processed for user `5a076dbe4d7e42d99c5c895885517fa4`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:08:50.666946 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-cb475812-9d5d-4488-9397-6ac8a41c905e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:50.678862 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6eef3dc-e7cc-4173-b028-807d8b7cd6d5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:50.680549 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6eef3dc-e7cc-4173-b028-807d8b7cd6d5 service cinder] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:50.688599 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:50.692330 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1260/2518] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:08:50 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 159 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 20 09:08:50.745762 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1259/2519] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:08:50 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 20 09:08:50.869076 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6eef3dc-e7cc-4173-b028-807d8b7cd6d5 service cinder] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:50.870563 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-d6eef3dc-e7cc-4173-b028-807d8b7cd6d5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:50.884119 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6eef3dc-e7cc-4173-b028-807d8b7cd6d5 service cinder] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:50.940979 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5093e0c3-7157-4667-a0e4-1fd8b03fe093 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:50.941751 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5093e0c3-7157-4667-a0e4-1fd8b03fe093 service cinder] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:50.942028 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5093e0c3-7157-4667-a0e4-1fd8b03fe093 service cinder] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:50.942302 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5093e0c3-7157-4667-a0e4-1fd8b03fe093 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:51.002128 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5093e0c3-7157-4667-a0e4-1fd8b03fe093 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:51.003618 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5093e0c3-7157-4667-a0e4-1fd8b03fe093 service cinder] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:51.015323 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1261/2520] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:08:50 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 147 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:08:51.290494 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb475812-9d5d-4488-9397-6ac8a41c905e None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:51.303713 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb475812-9d5d-4488-9397-6ac8a41c905e None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:51.307471 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-063cf802-f2f3-4e3c-a111-e5709cad9a39 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:51.308079 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-063cf802-f2f3-4e3c-a111-e5709cad9a39 service cinder] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:51.308384 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-063cf802-f2f3-4e3c-a111-e5709cad9a39 service cinder] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:51.308653 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-063cf802-f2f3-4e3c-a111-e5709cad9a39 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:51.310441 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-063cf802-f2f3-4e3c-a111-e5709cad9a39 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:51.379050 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-063cf802-f2f3-4e3c-a111-e5709cad9a39 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:51.380428 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-063cf802-f2f3-4e3c-a111-e5709cad9a39 service cinder] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:51.390303 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1260/2521] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:08:51 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:08:58.834978 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5093e0c3-7157-4667-a0e4-1fd8b03fe093 service cinder] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:58.866249 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=100 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:58.898085 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5093e0c3-7157-4667-a0e4-1fd8b03fe093 service cinder] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:58.902605 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f18f0e37-cd76-4e8b-baab-876a18c34686 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:58.903212 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18f0e37-cd76-4e8b-baab-876a18c34686 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:58.903486 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18f0e37-cd76-4e8b-baab-876a18c34686 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:58.903747 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18f0e37-cd76-4e8b-baab-876a18c34686 admin admin] PATH_INFO: `/v3/users/1b7f770792fb46dcb31954ad136ea64a` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:58.904932 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f18f0e37-cd76-4e8b-baab-876a18c34686 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1b7f770792fb46dcb31954ad136ea64a)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:58.906937 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f18f0e37-cd76-4e8b-baab-876a18c34686 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:58.955697 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-f18f0e37-cd76-4e8b-baab-876a18c34686 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1b7f770792fb46dcb31954ad136ea64a'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:58.964032 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-f18f0e37-cd76-4e8b-baab-876a18c34686 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1b7f770792fb46dcb31954ad136ea64a'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:58.976574 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-f18f0e37-cd76-4e8b-baab-876a18c34686 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1b7f770792fb46dcb31954ad136ea64a'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:58.983897 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-f18f0e37-cd76-4e8b-baab-876a18c34686 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1b7f770792fb46dcb31954ad136ea64a'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:58.984543 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-f18f0e37-cd76-4e8b-baab-876a18c34686 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1b7f770792fb46dcb31954ad136ea64a'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:58.990393 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1262/2522] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:08:58 2022] DELETE /identity/v3/users/1b7f770792fb46dcb31954ad136ea64a => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:08:58.994816 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec8562c4-af65-41bd-8ece-e8aa0609d716 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:58.995306 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec8562c4-af65-41bd-8ece-e8aa0609d716 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:58.995426 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec8562c4-af65-41bd-8ece-e8aa0609d716 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:59.014912 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-ec8562c4-af65-41bd-8ece-e8aa0609d716 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:08:59.016099 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-ec8562c4-af65-41bd-8ece-e8aa0609d716 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:59.074814 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1261/2523] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:08:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:08:59.083706 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f18f0e37-cd76-4e8b-baab-876a18c34686 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:59.085149 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-f18f0e37-cd76-4e8b-baab-876a18c34686 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:59.160414 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f18f0e37-cd76-4e8b-baab-876a18c34686 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:59.163971 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e9ad464-3a61-4138-84f8-eaacbb4865cc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:59.164555 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e9ad464-3a61-4138-84f8-eaacbb4865cc service neutron] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:59.164828 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e9ad464-3a61-4138-84f8-eaacbb4865cc service neutron] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:59.165094 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e9ad464-3a61-4138-84f8-eaacbb4865cc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:59.174342 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e9ad464-3a61-4138-84f8-eaacbb4865cc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:59.175764 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e9ad464-3a61-4138-84f8-eaacbb4865cc service neutron] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:59.185633 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1263/2524] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:08:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:08:59.461795 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec8562c4-af65-41bd-8ece-e8aa0609d716 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:59.463358 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-ec8562c4-af65-41bd-8ece-e8aa0609d716 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:59.470434 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e9ad464-3a61-4138-84f8-eaacbb4865cc service neutron] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:59.472337 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-2e9ad464-3a61-4138-84f8-eaacbb4865cc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:59.478565 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec8562c4-af65-41bd-8ece-e8aa0609d716 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:59.485839 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45147921-4da1-44db-90b7-90c6e9fe89e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:59.486496 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45147921-4da1-44db-90b7-90c6e9fe89e2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:59.486771 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45147921-4da1-44db-90b7-90c6e9fe89e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:59.487037 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45147921-4da1-44db-90b7-90c6e9fe89e2 admin admin] PATH_INFO: `/v3/projects/6ea45b923e70427ea00dc803fc8118ed` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:59.488145 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45147921-4da1-44db-90b7-90c6e9fe89e2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6ea45b923e70427ea00dc803fc8118ed)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:59.490783 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45147921-4da1-44db-90b7-90c6e9fe89e2 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:59.494443 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e9ad464-3a61-4138-84f8-eaacbb4865cc service neutron] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:59.507319 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50f76308-16c9-422e-ba6c-b005c9735aab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:59.507953 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f76308-16c9-422e-ba6c-b005c9735aab admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:59.508235 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f76308-16c9-422e-ba6c-b005c9735aab admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:59.508511 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f76308-16c9-422e-ba6c-b005c9735aab admin admin] PATH_INFO: `/v3/users/5bd3d936dcaf4fd8926e93b17b38114c` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:59.509701 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50f76308-16c9-422e-ba6c-b005c9735aab admin admin] RBAC: Authorizing `identity:delete_user(user_id=5bd3d936dcaf4fd8926e93b17b38114c)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:59.511702 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50f76308-16c9-422e-ba6c-b005c9735aab admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:59.541688 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:59.544298 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:59.547691 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-45147921-4da1-44db-90b7-90c6e9fe89e2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6ea45b923e70427ea00dc803fc8118ed', 'request_id': 'req-45147921-4da1-44db-90b7-90c6e9fe89e2'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:59.556013 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-45147921-4da1-44db-90b7-90c6e9fe89e2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6ea45b923e70427ea00dc803fc8118ed', 'request_id': 'req-45147921-4da1-44db-90b7-90c6e9fe89e2'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:59.562161 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-45147921-4da1-44db-90b7-90c6e9fe89e2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6ea45b923e70427ea00dc803fc8118ed', 'request_id': 'req-45147921-4da1-44db-90b7-90c6e9fe89e2'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:59.565703 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-50f76308-16c9-422e-ba6c-b005c9735aab admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5bd3d936dcaf4fd8926e93b17b38114c'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:59.573124 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-50f76308-16c9-422e-ba6c-b005c9735aab admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5bd3d936dcaf4fd8926e93b17b38114c'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:59.576851 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-45147921-4da1-44db-90b7-90c6e9fe89e2 admin admin] The token cache is being invalidate because project 6ea45b923e70427ea00dc803fc8118ed was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81561) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:08:59.577158 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-45147921-4da1-44db-90b7-90c6e9fe89e2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:59.577779 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-45147921-4da1-44db-90b7-90c6e9fe89e2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:59.579568 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1262/2525] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:08:59 2022] DELETE /identity/v3/projects/6ea45b923e70427ea00dc803fc8118ed => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:08:59.583213 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45147921-4da1-44db-90b7-90c6e9fe89e2 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:59.584447 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-45147921-4da1-44db-90b7-90c6e9fe89e2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:59.585548 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-50f76308-16c9-422e-ba6c-b005c9735aab admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5bd3d936dcaf4fd8926e93b17b38114c'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:59.593275 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-50f76308-16c9-422e-ba6c-b005c9735aab admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5bd3d936dcaf4fd8926e93b17b38114c'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:59.593971 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-50f76308-16c9-422e-ba6c-b005c9735aab admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5bd3d936dcaf4fd8926e93b17b38114c'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:59.600110 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1264/2526] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:08:59 2022] DELETE /identity/v3/users/5bd3d936dcaf4fd8926e93b17b38114c => generated 0 bytes in 130 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:08:59.604307 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03b8912-c15a-41b8-a1a9-329a27f315e2 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:59.604642 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03b8912-c15a-41b8-a1a9-329a27f315e2 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:59.604910 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03b8912-c15a-41b8-a1a9-329a27f315e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:59.623799 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-d03b8912-c15a-41b8-a1a9-329a27f315e2 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:08:59.624977 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-d03b8912-c15a-41b8-a1a9-329a27f315e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:59.654738 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45147921-4da1-44db-90b7-90c6e9fe89e2 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:59.689341 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1265/2527] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:08:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 20 09:08:59.696100 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d03b8912-c15a-41b8-a1a9-329a27f315e2 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:08:59.697618 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-d03b8912-c15a-41b8-a1a9-329a27f315e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:08:59.721302 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b440043-7156-4567-965a-1f5c391a3b1f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:59.721931 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b440043-7156-4567-965a-1f5c391a3b1f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:59.722217 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b440043-7156-4567-965a-1f5c391a3b1f admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:59.722490 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b440043-7156-4567-965a-1f5c391a3b1f admin admin] PATH_INFO: `/v3/users/5a076dbe4d7e42d99c5c895885517fa4` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:59.723605 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b440043-7156-4567-965a-1f5c391a3b1f admin admin] RBAC: Authorizing `identity:delete_user(user_id=5a076dbe4d7e42d99c5c895885517fa4)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:59.725531 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b440043-7156-4567-965a-1f5c391a3b1f admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:59.757852 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d03b8912-c15a-41b8-a1a9-329a27f315e2 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:08:59.762205 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67c38b20-f6bc-4bed-aaa5-5a170dc25e0e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:08:59.762829 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c38b20-f6bc-4bed-aaa5-5a170dc25e0e service neutron] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:08:59.763106 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c38b20-f6bc-4bed-aaa5-5a170dc25e0e service neutron] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:08:59.763504 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c38b20-f6bc-4bed-aaa5-5a170dc25e0e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:08:59.773187 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67c38b20-f6bc-4bed-aaa5-5a170dc25e0e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:08:59.773655 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-3b440043-7156-4567-965a-1f5c391a3b1f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5a076dbe4d7e42d99c5c895885517fa4'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:59.774680 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67c38b20-f6bc-4bed-aaa5-5a170dc25e0e service neutron] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:08:59.783786 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-3b440043-7156-4567-965a-1f5c391a3b1f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5a076dbe4d7e42d99c5c895885517fa4'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:59.785679 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1266/2528] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:08:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 20 09:08:59.797390 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-3b440043-7156-4567-965a-1f5c391a3b1f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5a076dbe4d7e42d99c5c895885517fa4'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:59.804286 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-3b440043-7156-4567-965a-1f5c391a3b1f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5a076dbe4d7e42d99c5c895885517fa4'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:59.805018 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-3b440043-7156-4567-965a-1f5c391a3b1f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5a076dbe4d7e42d99c5c895885517fa4'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:08:59.807293 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:DELETE=14,keystone:SELECT=84,keystone:INSERT=2 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:08:59.812762 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1263/2529] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:08:59 2022] DELETE /identity/v3/users/5a076dbe4d7e42d99c5c895885517fa4 => generated 0 bytes in 230 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:09:00.054529 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67c38b20-f6bc-4bed-aaa5-5a170dc25e0e service neutron] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:09:00.056175 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-67c38b20-f6bc-4bed-aaa5-5a170dc25e0e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:09:00.070343 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67c38b20-f6bc-4bed-aaa5-5a170dc25e0e service neutron] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:09:00.092925 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b440043-7156-4567-965a-1f5c391a3b1f admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:09:00.094795 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-3b440043-7156-4567-965a-1f5c391a3b1f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:09:00.124471 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:DELETE=6,keystone:SELECT=93,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:09:00.141843 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09c1e81c-130b-47aa-b75f-66dd6d1e7bcb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:09:00.142604 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c1e81c-130b-47aa-b75f-66dd6d1e7bcb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:09:00.142756 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c1e81c-130b-47aa-b75f-66dd6d1e7bcb admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:09:00.142993 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c1e81c-130b-47aa-b75f-66dd6d1e7bcb admin admin] PATH_INFO: `/v3/projects/05700a8ee8f942949117a3bf1643b00e` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:09:00.144131 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09c1e81c-130b-47aa-b75f-66dd6d1e7bcb admin admin] RBAC: Authorizing `identity:delete_project(project_id=05700a8ee8f942949117a3bf1643b00e)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:09:00.146043 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09c1e81c-130b-47aa-b75f-66dd6d1e7bcb admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:09:00.167483 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b440043-7156-4567-965a-1f5c391a3b1f admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:09:00.172672 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6c78501-04bc-4d6d-aab8-e645510ac041 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:09:00.173409 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c78501-04bc-4d6d-aab8-e645510ac041 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:09:00.173825 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c78501-04bc-4d6d-aab8-e645510ac041 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:09:00.174212 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c78501-04bc-4d6d-aab8-e645510ac041 admin admin] PATH_INFO: `/v3/projects/efeaf19187264177a120e64ea7d15e05` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:09:00.175421 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6c78501-04bc-4d6d-aab8-e645510ac041 admin admin] RBAC: Authorizing `identity:delete_project(project_id=efeaf19187264177a120e64ea7d15e05)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:09:00.177567 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6c78501-04bc-4d6d-aab8-e645510ac041 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:09:00.194511 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-09c1e81c-130b-47aa-b75f-66dd6d1e7bcb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '05700a8ee8f942949117a3bf1643b00e', 'request_id': 'req-09c1e81c-130b-47aa-b75f-66dd6d1e7bcb'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:09:00.201429 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-09c1e81c-130b-47aa-b75f-66dd6d1e7bcb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '05700a8ee8f942949117a3bf1643b00e', 'request_id': 'req-09c1e81c-130b-47aa-b75f-66dd6d1e7bcb'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:09:00.207184 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-09c1e81c-130b-47aa-b75f-66dd6d1e7bcb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '05700a8ee8f942949117a3bf1643b00e', 'request_id': 'req-09c1e81c-130b-47aa-b75f-66dd6d1e7bcb'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:09:00.221390 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-09c1e81c-130b-47aa-b75f-66dd6d1e7bcb admin admin] The token cache is being invalidate because project 05700a8ee8f942949117a3bf1643b00e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81562) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:09:00.221679 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-09c1e81c-130b-47aa-b75f-66dd6d1e7bcb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:09:00.222295 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-09c1e81c-130b-47aa-b75f-66dd6d1e7bcb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:09:00.223794 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1267/2530] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:09:00 2022] DELETE /identity/v3/projects/05700a8ee8f942949117a3bf1643b00e => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:09:00.227580 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09c1e81c-130b-47aa-b75f-66dd6d1e7bcb admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:09:00.228852 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-09c1e81c-130b-47aa-b75f-66dd6d1e7bcb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:09:00.230344 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-f6c78501-04bc-4d6d-aab8-e645510ac041 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'efeaf19187264177a120e64ea7d15e05', 'request_id': 'req-f6c78501-04bc-4d6d-aab8-e645510ac041'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:09:00.236832 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-f6c78501-04bc-4d6d-aab8-e645510ac041 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'efeaf19187264177a120e64ea7d15e05', 'request_id': 'req-f6c78501-04bc-4d6d-aab8-e645510ac041'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:09:00.244240 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-f6c78501-04bc-4d6d-aab8-e645510ac041 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'efeaf19187264177a120e64ea7d15e05', 'request_id': 'req-f6c78501-04bc-4d6d-aab8-e645510ac041'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:09:00.258580 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-f6c78501-04bc-4d6d-aab8-e645510ac041 admin admin] The token cache is being invalidate because project efeaf19187264177a120e64ea7d15e05 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81561) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:09:00.258964 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-f6c78501-04bc-4d6d-aab8-e645510ac041 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:09:00.262243 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-f6c78501-04bc-4d6d-aab8-e645510ac041 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:09:00.263903 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1264/2531] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:09:00 2022] DELETE /identity/v3/projects/efeaf19187264177a120e64ea7d15e05 => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:09:00.269329 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05ae7eff-961c-4fd1-bfdb-5c9d9c09d225 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:09:00.269700 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05ae7eff-961c-4fd1-bfdb-5c9d9c09d225 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:09:00.269972 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05ae7eff-961c-4fd1-bfdb-5c9d9c09d225 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:09:00.289000 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-05ae7eff-961c-4fd1-bfdb-5c9d9c09d225 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:09:00.290185 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-05ae7eff-961c-4fd1-bfdb-5c9d9c09d225 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:09:00.299538 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09c1e81c-130b-47aa-b75f-66dd6d1e7bcb admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:09:00.351520 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1265/2532] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:09:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 20 09:09:00.356288 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89e798ef-a303-4106-804b-b8fb681b1d93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:09:00.356997 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e798ef-a303-4106-804b-b8fb681b1d93 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:09:00.357814 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e798ef-a303-4106-804b-b8fb681b1d93 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:09:00.357814 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e798ef-a303-4106-804b-b8fb681b1d93 admin admin] PATH_INFO: `/v3/users/4f71a07a9eea473cbfa5aeba3134bf5a` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:09:00.358980 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89e798ef-a303-4106-804b-b8fb681b1d93 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4f71a07a9eea473cbfa5aeba3134bf5a)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:09:00.359688 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2315161-b1e9-4e80-93c2-bac879efa389 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:09:00.360053 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2315161-b1e9-4e80-93c2-bac879efa389 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:09:00.360246 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2315161-b1e9-4e80-93c2-bac879efa389 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:09:00.361378 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89e798ef-a303-4106-804b-b8fb681b1d93 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:09:00.380725 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-d2315161-b1e9-4e80-93c2-bac879efa389 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:09:00.381981 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-d2315161-b1e9-4e80-93c2-bac879efa389 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:09:00.392480 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1266/2533] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:09:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:09:00.396601 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:09:00.397908 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2315161-b1e9-4e80-93c2-bac879efa389 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:09:00.407338 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2315161-b1e9-4e80-93c2-bac879efa389 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:09:00.412150 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7cddf44-2803-45d0-a07b-93eff268cc4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:09:00.412747 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7cddf44-2803-45d0-a07b-93eff268cc4b admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:09:00.413056 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7cddf44-2803-45d0-a07b-93eff268cc4b admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:09:00.413325 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7cddf44-2803-45d0-a07b-93eff268cc4b admin admin] PATH_INFO: `/v3/domains` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:09:00.413967 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7cddf44-2803-45d0-a07b-93eff268cc4b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81561) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 09:09:00.414266 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7cddf44-2803-45d0-a07b-93eff268cc4b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:09:00.415904 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7cddf44-2803-45d0-a07b-93eff268cc4b admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:09:00.417747 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-89e798ef-a303-4106-804b-b8fb681b1d93 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4f71a07a9eea473cbfa5aeba3134bf5a'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:09:00.428107 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-89e798ef-a303-4106-804b-b8fb681b1d93 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4f71a07a9eea473cbfa5aeba3134bf5a'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:09:00.432807 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1267/2534] 10.222.0.36 () {58 vars in 1153 bytes} [Thu Oct 20 09:09:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 09:09:00.439018 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7cddf44-2803-45d0-a07b-93eff268cc4b admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:09:00.445839 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-89e798ef-a303-4106-804b-b8fb681b1d93 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4f71a07a9eea473cbfa5aeba3134bf5a'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:09:00.450181 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7cddf44-2803-45d0-a07b-93eff268cc4b admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:09:00.453488 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-89e798ef-a303-4106-804b-b8fb681b1d93 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4f71a07a9eea473cbfa5aeba3134bf5a'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:09:00.454262 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-89e798ef-a303-4106-804b-b8fb681b1d93 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4f71a07a9eea473cbfa5aeba3134bf5a'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:09:00.455010 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a98256f6-42a5-435f-b816-d9cbf77f6bd2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:09:00.455660 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a98256f6-42a5-435f-b816-d9cbf77f6bd2 admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:09:00.456031 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a98256f6-42a5-435f-b816-d9cbf77f6bd2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:09:00.456355 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a98256f6-42a5-435f-b816-d9cbf77f6bd2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:09:00.457046 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a98256f6-42a5-435f-b816-d9cbf77f6bd2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:09:00.459053 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a98256f6-42a5-435f-b816-d9cbf77f6bd2 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:09:00.460285 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1268/2535] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:09:00 2022] DELETE /identity/v3/users/4f71a07a9eea473cbfa5aeba3134bf5a => generated 0 bytes in 233 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:09:00.472088 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:09:00.492079 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1268/2536] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:09:00 2022] POST /identity/v3/projects => generated 367 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 20 09:09:00.497043 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89e798ef-a303-4106-804b-b8fb681b1d93 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:09:00.498463 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-89e798ef-a303-4106-804b-b8fb681b1d93 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:09:00.568763 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89e798ef-a303-4106-804b-b8fb681b1d93 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:09:00.631758 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff7a27c-465f-4782-b35f-52cca0c44407 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:09:00.632368 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff7a27c-465f-4782-b35f-52cca0c44407 admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:09:00.632641 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff7a27c-465f-4782-b35f-52cca0c44407 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:09:00.632913 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff7a27c-465f-4782-b35f-52cca0c44407 admin admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:09:00.633544 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ff7a27c-465f-4782-b35f-52cca0c44407 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:09:00.635481 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ff7a27c-465f-4782-b35f-52cca0c44407 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:09:00.657755 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1269/2537] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:09:00 2022] POST /identity/v3/users => generated 284 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:09:00.661871 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a98256f6-42a5-435f-b816-d9cbf77f6bd2 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:09:00.673216 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a98256f6-42a5-435f-b816-d9cbf77f6bd2 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:09:00.677942 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-295c4537-c804-4ac3-9a25-bc76e80e69e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:09:00.678569 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295c4537-c804-4ac3-9a25-bc76e80e69e5 admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:09:00.678850 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295c4537-c804-4ac3-9a25-bc76e80e69e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:09:00.679146 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295c4537-c804-4ac3-9a25-bc76e80e69e5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:09:00.679781 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-295c4537-c804-4ac3-9a25-bc76e80e69e5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:09:00.681384 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-295c4537-c804-4ac3-9a25-bc76e80e69e5 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:09:00.692120 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1269/2538] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:09:00 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:09:00.696054 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff7a27c-465f-4782-b35f-52cca0c44407 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:09:00.707351 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff7a27c-465f-4782-b35f-52cca0c44407 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:09:00.713197 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab67451b-86a2-48d2-803e-18f6d4475b60 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:09:00.713914 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab67451b-86a2-48d2-803e-18f6d4475b60 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:09:00.714284 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab67451b-86a2-48d2-803e-18f6d4475b60 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:09:00.714664 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab67451b-86a2-48d2-803e-18f6d4475b60 admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:09:00.715725 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab67451b-86a2-48d2-803e-18f6d4475b60 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:09:00.717568 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab67451b-86a2-48d2-803e-18f6d4475b60 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:09:00.728627 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1270/2539] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:09:00 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 20 09:09:00.733381 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-295c4537-c804-4ac3-9a25-bc76e80e69e5 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:09:00.742873 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab67451b-86a2-48d2-803e-18f6d4475b60 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:09:00.744381 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-295c4537-c804-4ac3-9a25-bc76e80e69e5 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:09:00.745127 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-ab67451b-86a2-48d2-803e-18f6d4475b60 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:09:00.749844 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4339cfd-e312-4add-a8c3-7312a5642f0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:09:00.750519 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4339cfd-e312-4add-a8c3-7312a5642f0d admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:09:00.750796 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4339cfd-e312-4add-a8c3-7312a5642f0d admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:09:00.751069 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4339cfd-e312-4add-a8c3-7312a5642f0d admin admin] PATH_INFO: `/v3/projects/cac8599536e941948e67aeb426a6ed27/users/b6af827029064fa986b65b71c6d37501/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:09:00.764587 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab67451b-86a2-48d2-803e-18f6d4475b60 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:09:00.768200 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4339cfd-e312-4add-a8c3-7312a5642f0d admin admin] RBAC: Authorizing `identity:create_grant(project_id=cac8599536e941948e67aeb426a6ed27, user_id=b6af827029064fa986b65b71c6d37501, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:09:00.770196 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e3d2691-c02c-4739-845c-ef45a7e19eba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:09:00.770944 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4339cfd-e312-4add-a8c3-7312a5642f0d admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:09:00.771052 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e3d2691-c02c-4739-845c-ef45a7e19eba admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:09:00.771255 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e3d2691-c02c-4739-845c-ef45a7e19eba admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:09:00.771538 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e3d2691-c02c-4739-845c-ef45a7e19eba admin admin] PATH_INFO: `/v3/projects/1f2e26cf7cf74c4e9c99f42d0740f914` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:09:00.772620 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e3d2691-c02c-4739-845c-ef45a7e19eba admin admin] RBAC: Authorizing `identity:delete_project(project_id=1f2e26cf7cf74c4e9c99f42d0740f914)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:09:00.774639 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e3d2691-c02c-4739-845c-ef45a7e19eba admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:09:00.786050 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1270/2540] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:09:00 2022] PUT /identity/v3/projects/cac8599536e941948e67aeb426a6ed27/users/b6af827029064fa986b65b71c6d37501/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:09:00.798050 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da89ffa0-0c6b-4c21-b83e-0f40b452c80d None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:09:00.798586 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da89ffa0-0c6b-4c21-b83e-0f40b452c80d None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:09:00.799026 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da89ffa0-0c6b-4c21-b83e-0f40b452c80d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:09:00.804042 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:DELETE=7,keystone:SELECT=89,keystone:INSERT=4 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:09:00.828476 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-8e3d2691-c02c-4739-845c-ef45a7e19eba admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1f2e26cf7cf74c4e9c99f42d0740f914', 'request_id': 'req-8e3d2691-c02c-4739-845c-ef45a7e19eba'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:09:00.830621 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-da89ffa0-0c6b-4c21-b83e-0f40b452c80d None None] MFA Rules not processed for user `b6af827029064fa986b65b71c6d37501`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:09:00.831862 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-da89ffa0-0c6b-4c21-b83e-0f40b452c80d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:09:00.835351 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-8e3d2691-c02c-4739-845c-ef45a7e19eba admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1f2e26cf7cf74c4e9c99f42d0740f914', 'request_id': 'req-8e3d2691-c02c-4739-845c-ef45a7e19eba'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:09:00.842371 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-8e3d2691-c02c-4739-845c-ef45a7e19eba admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1f2e26cf7cf74c4e9c99f42d0740f914', 'request_id': 'req-8e3d2691-c02c-4739-845c-ef45a7e19eba'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:09:00.856815 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-8e3d2691-c02c-4739-845c-ef45a7e19eba admin admin] The token cache is being invalidate because project 1f2e26cf7cf74c4e9c99f42d0740f914 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81562) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:09:00.857100 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-8e3d2691-c02c-4739-845c-ef45a7e19eba admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:09:00.857751 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-8e3d2691-c02c-4739-845c-ef45a7e19eba admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:09:00.859592 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1271/2541] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:09:00 2022] DELETE /identity/v3/projects/1f2e26cf7cf74c4e9c99f42d0740f914 => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:09:00.862969 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e3d2691-c02c-4739-845c-ef45a7e19eba admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:09:00.864277 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-8e3d2691-c02c-4739-845c-ef45a7e19eba admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:09:00.910342 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1271/2542] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:09:00 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 20 09:09:00.923040 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da89ffa0-0c6b-4c21-b83e-0f40b452c80d None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:09:00.924479 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-da89ffa0-0c6b-4c21-b83e-0f40b452c80d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:09:00.933867 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e3d2691-c02c-4739-845c-ef45a7e19eba admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:09:00.938634 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b44594-26e9-4598-8d49-88327b082008 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:09:00.939330 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b44594-26e9-4598-8d49-88327b082008 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:09:00.939624 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b44594-26e9-4598-8d49-88327b082008 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:09:00.939871 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b44594-26e9-4598-8d49-88327b082008 admin admin] PATH_INFO: `/v3/users/6d5202a0f4db4cb28d7505120ee8715a` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:09:00.940988 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3b44594-26e9-4598-8d49-88327b082008 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6d5202a0f4db4cb28d7505120ee8715a)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:09:00.943016 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3b44594-26e9-4598-8d49-88327b082008 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:09:00.984174 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da89ffa0-0c6b-4c21-b83e-0f40b452c80d None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:09:00.987895 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f1328a3-f3fb-4586-a99f-ac653aa60274 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce043914cd9042909f70ab0adb272d34', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:09:00.988497 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f1328a3-f3fb-4586-a99f-ac653aa60274 service glance] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:09:00.988774 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f1328a3-f3fb-4586-a99f-ac653aa60274 service glance] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:09:00.989042 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f1328a3-f3fb-4586-a99f-ac653aa60274 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:09:00.990448 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-a3b44594-26e9-4598-8d49-88327b082008 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6d5202a0f4db4cb28d7505120ee8715a'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:09:00.997949 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f1328a3-f3fb-4586-a99f-ac653aa60274 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:09:00.998115 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-a3b44594-26e9-4598-8d49-88327b082008 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6d5202a0f4db4cb28d7505120ee8715a'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:09:00.999403 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f1328a3-f3fb-4586-a99f-ac653aa60274 service glance] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:09:01.012392 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1272/2543] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:09:00 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 20 09:09:01.012596 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-a3b44594-26e9-4598-8d49-88327b082008 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6d5202a0f4db4cb28d7505120ee8715a'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:09:01.019879 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-a3b44594-26e9-4598-8d49-88327b082008 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6d5202a0f4db4cb28d7505120ee8715a'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:09:01.020021 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-a3b44594-26e9-4598-8d49-88327b082008 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6d5202a0f4db4cb28d7505120ee8715a'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:09:01.025815 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1272/2544] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:09:00 2022] DELETE /identity/v3/users/6d5202a0f4db4cb28d7505120ee8715a => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:09:01.322020 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f1328a3-f3fb-4586-a99f-ac653aa60274 service glance] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:09:01.334043 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f1328a3-f3fb-4586-a99f-ac653aa60274 service glance] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:09:01.389289 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:09:01.404128 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-138f4967-025e-4bf5-98e2-d549d778f5a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:09:01.404965 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-138f4967-025e-4bf5-98e2-d549d778f5a4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:09:01.405244 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-138f4967-025e-4bf5-98e2-d549d778f5a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:09:01.405513 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-138f4967-025e-4bf5-98e2-d549d778f5a4 admin admin] PATH_INFO: `/v3/projects/f6b5cfda167247ef89d9e4139967492b` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:09:01.406884 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-138f4967-025e-4bf5-98e2-d549d778f5a4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f6b5cfda167247ef89d9e4139967492b)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:09:01.409125 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-138f4967-025e-4bf5-98e2-d549d778f5a4 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:09:01.460119 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-138f4967-025e-4bf5-98e2-d549d778f5a4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f6b5cfda167247ef89d9e4139967492b', 'request_id': 'req-138f4967-025e-4bf5-98e2-d549d778f5a4'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:09:01.468798 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-138f4967-025e-4bf5-98e2-d549d778f5a4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f6b5cfda167247ef89d9e4139967492b', 'request_id': 'req-138f4967-025e-4bf5-98e2-d549d778f5a4'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:09:01.476184 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-138f4967-025e-4bf5-98e2-d549d778f5a4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f6b5cfda167247ef89d9e4139967492b', 'request_id': 'req-138f4967-025e-4bf5-98e2-d549d778f5a4'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:09:01.496799 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-138f4967-025e-4bf5-98e2-d549d778f5a4 admin admin] The token cache is being invalidate because project f6b5cfda167247ef89d9e4139967492b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81561) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:09:01.497299 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-138f4967-025e-4bf5-98e2-d549d778f5a4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:09:01.498144 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-138f4967-025e-4bf5-98e2-d549d778f5a4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:09:01.499959 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1273/2545] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:09:01 2022] DELETE /identity/v3/projects/f6b5cfda167247ef89d9e4139967492b => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:09:01.506458 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf74611-6804-4b74-9465-8e18dfbe0efc None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:09:01.507849 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf74611-6804-4b74-9465-8e18dfbe0efc None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:09:01.508835 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf74611-6804-4b74-9465-8e18dfbe0efc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:09:01.530053 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-cdf74611-6804-4b74-9465-8e18dfbe0efc None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:09:01.531291 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-cdf74611-6804-4b74-9465-8e18dfbe0efc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:09:01.557728 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=82,keystone:DELETE=16,keystone:INSERT=2 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:09:01.592890 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1273/2546] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:09:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:09:01.600624 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7fc1e9-8f0b-489f-9d8e-83a8ce1350b6 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:09:01.600911 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7fc1e9-8f0b-489f-9d8e-83a8ce1350b6 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:09:01.601186 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7fc1e9-8f0b-489f-9d8e-83a8ce1350b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:09:01.621422 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-cc7fc1e9-8f0b-489f-9d8e-83a8ce1350b6 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:09:01.622655 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-cc7fc1e9-8f0b-489f-9d8e-83a8ce1350b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:09:01.630388 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1274/2547] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:09:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:09:01.634582 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf74611-6804-4b74-9465-8e18dfbe0efc None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:09:01.644660 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf74611-6804-4b74-9465-8e18dfbe0efc None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:09:01.649412 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-158e17c1-306e-4cf9-8cc7-895d9b7862bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:09:01.650001 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158e17c1-306e-4cf9-8cc7-895d9b7862bb admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:09:01.650325 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158e17c1-306e-4cf9-8cc7-895d9b7862bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:09:01.650662 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158e17c1-306e-4cf9-8cc7-895d9b7862bb admin admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:09:01.651608 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-158e17c1-306e-4cf9-8cc7-895d9b7862bb admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 09:09:01.651966 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-158e17c1-306e-4cf9-8cc7-895d9b7862bb admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:09:01.653857 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-158e17c1-306e-4cf9-8cc7-895d9b7862bb admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:09:01.665411 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1274/2548] 10.222.0.36 () {58 vars in 1153 bytes} [Thu Oct 20 09:09:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 20 09:09:01.669792 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc7fc1e9-8f0b-489f-9d8e-83a8ce1350b6 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:09:01.679261 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc7fc1e9-8f0b-489f-9d8e-83a8ce1350b6 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:09:01.684427 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23817791-67e1-4c87-a951-3ac92142a812 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:09:01.685015 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23817791-67e1-4c87-a951-3ac92142a812 admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:09:01.685291 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23817791-67e1-4c87-a951-3ac92142a812 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:09:01.685585 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23817791-67e1-4c87-a951-3ac92142a812 admin admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:09:01.686217 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23817791-67e1-4c87-a951-3ac92142a812 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:09:01.688154 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23817791-67e1-4c87-a951-3ac92142a812 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:09:01.717191 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1275/2549] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:09:01 2022] POST /identity/v3/projects => generated 375 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:09:01.721304 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-158e17c1-306e-4cf9-8cc7-895d9b7862bb admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:09:01.732175 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-158e17c1-306e-4cf9-8cc7-895d9b7862bb admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:09:01.803219 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6acb51a3-0469-4095-bf1c-a71b17454b52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:09:01.803828 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6acb51a3-0469-4095-bf1c-a71b17454b52 admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:09:01.804104 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6acb51a3-0469-4095-bf1c-a71b17454b52 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:09:01.804375 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6acb51a3-0469-4095-bf1c-a71b17454b52 admin admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:09:01.805000 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6acb51a3-0469-4095-bf1c-a71b17454b52 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:09:01.807083 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6acb51a3-0469-4095-bf1c-a71b17454b52 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:09:01.830910 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1275/2550] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:09:01 2022] POST /identity/v3/users => generated 288 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:09:01.835162 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23817791-67e1-4c87-a951-3ac92142a812 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:09:01.847408 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23817791-67e1-4c87-a951-3ac92142a812 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:09:01.852913 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fba0c71-746e-4fc9-9918-ea126fadb36c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:09:01.853520 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fba0c71-746e-4fc9-9918-ea126fadb36c admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:09:01.853800 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fba0c71-746e-4fc9-9918-ea126fadb36c admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:09:01.854062 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fba0c71-746e-4fc9-9918-ea126fadb36c admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:09:01.854751 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fba0c71-746e-4fc9-9918-ea126fadb36c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:09:01.856455 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fba0c71-746e-4fc9-9918-ea126fadb36c admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:09:01.870566 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1276/2551] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:09:01 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:09:01.875871 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6acb51a3-0469-4095-bf1c-a71b17454b52 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:09:01.886216 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6acb51a3-0469-4095-bf1c-a71b17454b52 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:09:01.892101 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c34ab6ca-a050-4b54-b9a3-f709875f6f47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:09:01.892693 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34ab6ca-a050-4b54-b9a3-f709875f6f47 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:09:01.892967 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34ab6ca-a050-4b54-b9a3-f709875f6f47 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:09:01.893233 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34ab6ca-a050-4b54-b9a3-f709875f6f47 admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:09:01.893906 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c34ab6ca-a050-4b54-b9a3-f709875f6f47 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:09:01.896261 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c34ab6ca-a050-4b54-b9a3-f709875f6f47 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:09:01.910411 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1276/2552] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:09:01 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 20 09:09:01.915027 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fba0c71-746e-4fc9-9918-ea126fadb36c admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:09:01.927207 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fba0c71-746e-4fc9-9918-ea126fadb36c admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:09:01.932222 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87ba69b4-3a0f-4bbf-b85c-5798fd9c4944 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:09:01.932895 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87ba69b4-3a0f-4bbf-b85c-5798fd9c4944 admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:09:01.933858 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87ba69b4-3a0f-4bbf-b85c-5798fd9c4944 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:09:01.934132 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87ba69b4-3a0f-4bbf-b85c-5798fd9c4944 admin admin] PATH_INFO: `/v3/projects/0ce71f46acb24cd899e0d08974c825ab/users/62af90b699ca42a4aa0ac8f0edfdfd31/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:09:01.951277 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87ba69b4-3a0f-4bbf-b85c-5798fd9c4944 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0ce71f46acb24cd899e0d08974c825ab, user_id=62af90b699ca42a4aa0ac8f0edfdfd31, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:09:01.954256 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87ba69b4-3a0f-4bbf-b85c-5798fd9c4944 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:09:01.971469 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1277/2553] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:09:01 2022] PUT /identity/v3/projects/0ce71f46acb24cd899e0d08974c825ab/users/62af90b699ca42a4aa0ac8f0edfdfd31/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:09:01.982185 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d289d8-698a-4c04-af66-3c9053a7468f None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:09:01.982492 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d289d8-698a-4c04-af66-3c9053a7468f None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:09:01.982772 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d289d8-698a-4c04-af66-3c9053a7468f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:09:02.015584 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-92d289d8-698a-4c04-af66-3c9053a7468f None None] MFA Rules not processed for user `62af90b699ca42a4aa0ac8f0edfdfd31`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:09:02.016794 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-92d289d8-698a-4c04-af66-3c9053a7468f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:09:02.097827 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1277/2554] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:09:01 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:09:02.109245 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87ba69b4-3a0f-4bbf-b85c-5798fd9c4944 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:09:02.109245 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-87ba69b4-3a0f-4bbf-b85c-5798fd9c4944 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:09:02.191437 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87ba69b4-3a0f-4bbf-b85c-5798fd9c4944 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:09:02.196646 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3922e283-dc42-488f-aff9-a7b79f7ab401 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:09:02.197442 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3922e283-dc42-488f-aff9-a7b79f7ab401 service cinder] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:09:02.197820 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3922e283-dc42-488f-aff9-a7b79f7ab401 service cinder] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:09:02.198157 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3922e283-dc42-488f-aff9-a7b79f7ab401 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:09:02.204757 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:09:02.208027 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3922e283-dc42-488f-aff9-a7b79f7ab401 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:09:02.210024 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3922e283-dc42-488f-aff9-a7b79f7ab401 service cinder] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:09:02.220705 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1278/2555] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:09:02 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:09:03.612096 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c88a620e-9122-4815-97db-823295db2700 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:09:03.612561 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c88a620e-9122-4815-97db-823295db2700 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:09:03.612658 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c88a620e-9122-4815-97db-823295db2700 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:09:03.639663 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-c88a620e-9122-4815-97db-823295db2700 None None] MFA Rules not processed for user `62af90b699ca42a4aa0ac8f0edfdfd31`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:09:03.641095 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-c88a620e-9122-4815-97db-823295db2700 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:09:03.654315 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1278/2556] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:09:03 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:09:05.662977 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3922e283-dc42-488f-aff9-a7b79f7ab401 service cinder] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:09:05.664382 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-3922e283-dc42-488f-aff9-a7b79f7ab401 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:09:05.770778 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3922e283-dc42-488f-aff9-a7b79f7ab401 service cinder] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:09:05.770778 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f30aa9e7-b882-481c-8c49-16e19fd69dce service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ad03ab76b9c44b1cb369475afa25eddd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:09:05.770778 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30aa9e7-b882-481c-8c49-16e19fd69dce service placement] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:09:05.770778 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30aa9e7-b882-481c-8c49-16e19fd69dce service placement] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:09:05.771197 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30aa9e7-b882-481c-8c49-16e19fd69dce service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:09:05.771197 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-f30aa9e7-b882-481c-8c49-16e19fd69dce service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:09:05.782353 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f30aa9e7-b882-481c-8c49-16e19fd69dce service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:09:05.783909 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f30aa9e7-b882-481c-8c49-16e19fd69dce service placement] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:09:05.794466 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1279/2557] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:09:05 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 132 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:09:06.533124 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c88a620e-9122-4815-97db-823295db2700 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:09:06.542812 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c88a620e-9122-4815-97db-823295db2700 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:09:06.546815 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-419151d0-43c5-4e78-a39c-86d15f768ece service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:09:06.547430 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-419151d0-43c5-4e78-a39c-86d15f768ece service cinder] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:09:06.547702 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-419151d0-43c5-4e78-a39c-86d15f768ece service cinder] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:09:06.547965 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-419151d0-43c5-4e78-a39c-86d15f768ece service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:09:06.549674 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-419151d0-43c5-4e78-a39c-86d15f768ece service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:09:06.567469 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:09:06.614569 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-419151d0-43c5-4e78-a39c-86d15f768ece service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:09:06.616133 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-419151d0-43c5-4e78-a39c-86d15f768ece service cinder] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:09:06.627423 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1279/2558] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:09:06 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:09:06.934653 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f30aa9e7-b882-481c-8c49-16e19fd69dce service placement] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:09:06.936204 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-f30aa9e7-b882-481c-8c49-16e19fd69dce service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:09:07.006591 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f30aa9e7-b882-481c-8c49-16e19fd69dce service placement] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:09:07.011114 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1291a5b2-cb75-481f-a926-d32ef487f817 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:09:07.011738 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1291a5b2-cb75-481f-a926-d32ef487f817 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:09:07.012022 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1291a5b2-cb75-481f-a926-d32ef487f817 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:09:07.012296 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1291a5b2-cb75-481f-a926-d32ef487f817 admin admin] PATH_INFO: `/v3/users/aaef5a9a56d64c1badace8afee33f438` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:09:07.013433 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1291a5b2-cb75-481f-a926-d32ef487f817 admin admin] RBAC: Authorizing `identity:delete_user(user_id=aaef5a9a56d64c1badace8afee33f438)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:09:07.015394 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1291a5b2-cb75-481f-a926-d32ef487f817 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:09:07.071024 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-1291a5b2-cb75-481f-a926-d32ef487f817 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'aaef5a9a56d64c1badace8afee33f438'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:09:07.079436 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-1291a5b2-cb75-481f-a926-d32ef487f817 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'aaef5a9a56d64c1badace8afee33f438'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:09:07.090698 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:09:07.097527 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-1291a5b2-cb75-481f-a926-d32ef487f817 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'aaef5a9a56d64c1badace8afee33f438'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:09:07.106648 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-1291a5b2-cb75-481f-a926-d32ef487f817 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'aaef5a9a56d64c1badace8afee33f438'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:09:07.107384 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-1291a5b2-cb75-481f-a926-d32ef487f817 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'aaef5a9a56d64c1badace8afee33f438'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:09:07.113716 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1280/2559] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:09:06 2022] DELETE /identity/v3/users/aaef5a9a56d64c1badace8afee33f438 => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:09:07.118537 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05c67d1-f7cc-461c-9e79-f891f93450b2 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:09:07.119095 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05c67d1-f7cc-461c-9e79-f891f93450b2 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:09:07.119454 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05c67d1-f7cc-461c-9e79-f891f93450b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:09:07.139687 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-a05c67d1-f7cc-461c-9e79-f891f93450b2 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:09:07.140941 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-a05c67d1-f7cc-461c-9e79-f891f93450b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:09:07.209640 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1280/2560] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:09:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:09:07.223742 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1291a5b2-cb75-481f-a926-d32ef487f817 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:09:07.225330 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-1291a5b2-cb75-481f-a926-d32ef487f817 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:09:07.288530 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1291a5b2-cb75-481f-a926-d32ef487f817 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:09:07.292265 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5797a4c-7331-4dde-b4e2-abe15070b91e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:09:07.292876 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5797a4c-7331-4dde-b4e2-abe15070b91e service neutron] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:09:07.293156 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5797a4c-7331-4dde-b4e2-abe15070b91e service neutron] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:09:07.293430 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5797a4c-7331-4dde-b4e2-abe15070b91e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:09:07.302686 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5797a4c-7331-4dde-b4e2-abe15070b91e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:09:07.304315 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5797a4c-7331-4dde-b4e2-abe15070b91e service neutron] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:09:07.314413 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1281/2561] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:09:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:09:07.752432 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a05c67d1-f7cc-461c-9e79-f891f93450b2 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:09:07.753995 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-a05c67d1-f7cc-461c-9e79-f891f93450b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:09:07.778373 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a05c67d1-f7cc-461c-9e79-f891f93450b2 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:09:07.783261 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48712f84-f397-4614-9019-0ce42cb5b825 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:09:07.784051 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48712f84-f397-4614-9019-0ce42cb5b825 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:09:07.784469 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48712f84-f397-4614-9019-0ce42cb5b825 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:09:07.784857 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48712f84-f397-4614-9019-0ce42cb5b825 admin admin] PATH_INFO: `/v3/projects/ae904361755c44d9842b1ce7829bb4ce` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:09:07.786114 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48712f84-f397-4614-9019-0ce42cb5b825 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ae904361755c44d9842b1ce7829bb4ce)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:09:07.788654 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48712f84-f397-4614-9019-0ce42cb5b825 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:09:07.850630 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-48712f84-f397-4614-9019-0ce42cb5b825 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ae904361755c44d9842b1ce7829bb4ce', 'request_id': 'req-48712f84-f397-4614-9019-0ce42cb5b825'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:09:07.857953 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-48712f84-f397-4614-9019-0ce42cb5b825 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ae904361755c44d9842b1ce7829bb4ce', 'request_id': 'req-48712f84-f397-4614-9019-0ce42cb5b825'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:09:07.864262 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-48712f84-f397-4614-9019-0ce42cb5b825 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ae904361755c44d9842b1ce7829bb4ce', 'request_id': 'req-48712f84-f397-4614-9019-0ce42cb5b825'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:09:07.881773 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-48712f84-f397-4614-9019-0ce42cb5b825 admin admin] The token cache is being invalidate because project ae904361755c44d9842b1ce7829bb4ce was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81562) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:09:07.882122 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-48712f84-f397-4614-9019-0ce42cb5b825 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:09:07.882892 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-48712f84-f397-4614-9019-0ce42cb5b825 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:09:07.890060 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1281/2562] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:09:07 2022] DELETE /identity/v3/projects/ae904361755c44d9842b1ce7829bb4ce => generated 0 bytes in 133 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:09:14.588746 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9dfbf53-9dcb-40cd-b41c-fef6c893ebf7 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:09:14.589268 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9dfbf53-9dcb-40cd-b41c-fef6c893ebf7 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:09:14.589268 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9dfbf53-9dcb-40cd-b41c-fef6c893ebf7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:09:14.609397 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-a9dfbf53-9dcb-40cd-b41c-fef6c893ebf7 None None] MFA Rules not processed for user `65ca2eae6db24bb094afaacdfb5c1980`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:09:14.610643 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-a9dfbf53-9dcb-40cd-b41c-fef6c893ebf7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:09:14.667812 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1282/2563] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:09:14 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:09:17.877743 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=73,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:09:24.660736 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:DELETE=1,keystone:SELECT=57 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:09:30.024683 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48712f84-f397-4614-9019-0ce42cb5b825 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:09:30.026072 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-48712f84-f397-4614-9019-0ce42cb5b825 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:09:30.088835 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48712f84-f397-4614-9019-0ce42cb5b825 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:09:30.094124 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f46e62c1-223c-4457-b358-8b66deeaf3a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:09:30.094774 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f46e62c1-223c-4457-b358-8b66deeaf3a8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:09:30.095053 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f46e62c1-223c-4457-b358-8b66deeaf3a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:09:30.095324 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f46e62c1-223c-4457-b358-8b66deeaf3a8 admin admin] PATH_INFO: `/v3/users/b6af827029064fa986b65b71c6d37501` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:09:30.096430 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f46e62c1-223c-4457-b358-8b66deeaf3a8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b6af827029064fa986b65b71c6d37501)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:09:30.098334 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f46e62c1-223c-4457-b358-8b66deeaf3a8 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:09:30.143352 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-f46e62c1-223c-4457-b358-8b66deeaf3a8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b6af827029064fa986b65b71c6d37501'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:09:30.150284 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-f46e62c1-223c-4457-b358-8b66deeaf3a8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b6af827029064fa986b65b71c6d37501'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:09:30.162366 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-f46e62c1-223c-4457-b358-8b66deeaf3a8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b6af827029064fa986b65b71c6d37501'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:09:30.168990 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-f46e62c1-223c-4457-b358-8b66deeaf3a8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b6af827029064fa986b65b71c6d37501'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:09:30.169624 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-f46e62c1-223c-4457-b358-8b66deeaf3a8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b6af827029064fa986b65b71c6d37501'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:09:30.176024 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1282/2564] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:09:30 2022] DELETE /identity/v3/users/b6af827029064fa986b65b71c6d37501 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:09:30.180551 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92481f51-9f39-4787-a53a-793565758c73 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:09:30.181049 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92481f51-9f39-4787-a53a-793565758c73 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:09:30.181170 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92481f51-9f39-4787-a53a-793565758c73 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:09:30.200711 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-92481f51-9f39-4787-a53a-793565758c73 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:09:30.201918 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-92481f51-9f39-4787-a53a-793565758c73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:09:30.258595 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1283/2565] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:09:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:09:30.268068 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f46e62c1-223c-4457-b358-8b66deeaf3a8 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:09:30.269445 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-f46e62c1-223c-4457-b358-8b66deeaf3a8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:09:30.325223 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f46e62c1-223c-4457-b358-8b66deeaf3a8 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:09:30.328807 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0336c994-fad7-48c6-9eea-10dbf0054981 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:09:30.329394 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0336c994-fad7-48c6-9eea-10dbf0054981 service neutron] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:09:30.329667 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0336c994-fad7-48c6-9eea-10dbf0054981 service neutron] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:09:30.329933 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0336c994-fad7-48c6-9eea-10dbf0054981 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:09:30.338678 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0336c994-fad7-48c6-9eea-10dbf0054981 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:09:30.340353 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0336c994-fad7-48c6-9eea-10dbf0054981 service neutron] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:09:30.349945 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1283/2566] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:09:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:09:30.616305 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92481f51-9f39-4787-a53a-793565758c73 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:09:30.617762 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-92481f51-9f39-4787-a53a-793565758c73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:09:30.629988 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92481f51-9f39-4787-a53a-793565758c73 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:09:30.635333 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a04de7a-1980-44cb-97d6-6b543d909560 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:09:30.635972 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a04de7a-1980-44cb-97d6-6b543d909560 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:09:30.636250 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a04de7a-1980-44cb-97d6-6b543d909560 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:09:30.636523 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a04de7a-1980-44cb-97d6-6b543d909560 admin admin] PATH_INFO: `/v3/projects/cac8599536e941948e67aeb426a6ed27` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:09:30.638245 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a04de7a-1980-44cb-97d6-6b543d909560 admin admin] RBAC: Authorizing `identity:delete_project(project_id=cac8599536e941948e67aeb426a6ed27)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:09:30.640742 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a04de7a-1980-44cb-97d6-6b543d909560 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:09:30.695334 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-3a04de7a-1980-44cb-97d6-6b543d909560 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cac8599536e941948e67aeb426a6ed27', 'request_id': 'req-3a04de7a-1980-44cb-97d6-6b543d909560'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:09:30.701739 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-3a04de7a-1980-44cb-97d6-6b543d909560 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cac8599536e941948e67aeb426a6ed27', 'request_id': 'req-3a04de7a-1980-44cb-97d6-6b543d909560'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:09:30.707917 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-3a04de7a-1980-44cb-97d6-6b543d909560 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cac8599536e941948e67aeb426a6ed27', 'request_id': 'req-3a04de7a-1980-44cb-97d6-6b543d909560'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:09:30.723575 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-3a04de7a-1980-44cb-97d6-6b543d909560 admin admin] The token cache is being invalidate because project cac8599536e941948e67aeb426a6ed27 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81561) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:09:30.723867 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-3a04de7a-1980-44cb-97d6-6b543d909560 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:09:30.724507 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-3a04de7a-1980-44cb-97d6-6b543d909560 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:09:30.726053 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1284/2567] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:09:30 2022] DELETE /identity/v3/projects/cac8599536e941948e67aeb426a6ed27 => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:09:30.730915 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6047a4c5-ca01-45fc-97a7-a7f73ff507f2 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:09:30.731223 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6047a4c5-ca01-45fc-97a7-a7f73ff507f2 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:09:30.731493 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6047a4c5-ca01-45fc-97a7-a7f73ff507f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:09:30.752120 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-6047a4c5-ca01-45fc-97a7-a7f73ff507f2 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:09:30.753332 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-6047a4c5-ca01-45fc-97a7-a7f73ff507f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:09:30.770643 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:09:30.820918 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1284/2568] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:09:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:09:30.829343 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7467836-25a2-4be5-adfc-8ac88d69a039 None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:09:30.829633 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7467836-25a2-4be5-adfc-8ac88d69a039 None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:09:30.829911 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7467836-25a2-4be5-adfc-8ac88d69a039 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:09:30.850099 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-c7467836-25a2-4be5-adfc-8ac88d69a039 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:09:30.851335 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-c7467836-25a2-4be5-adfc-8ac88d69a039 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:09:30.859273 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1285/2569] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:09:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:09:30.863559 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6047a4c5-ca01-45fc-97a7-a7f73ff507f2 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:09:30.873271 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6047a4c5-ca01-45fc-97a7-a7f73ff507f2 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:09:30.878566 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-445ecedf-951d-48d8-a029-888151a4bc3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:09:30.879863 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-445ecedf-951d-48d8-a029-888151a4bc3c admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:09:30.880145 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-445ecedf-951d-48d8-a029-888151a4bc3c admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:09:30.880419 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-445ecedf-951d-48d8-a029-888151a4bc3c admin admin] PATH_INFO: `/v3/domains` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:09:30.881075 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-445ecedf-951d-48d8-a029-888151a4bc3c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81562) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 20 09:09:30.881382 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-445ecedf-951d-48d8-a029-888151a4bc3c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:09:30.883035 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-445ecedf-951d-48d8-a029-888151a4bc3c admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:09:30.898871 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1285/2570] 10.222.0.36 () {58 vars in 1153 bytes} [Thu Oct 20 09:09:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 20 09:09:30.899979 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7467836-25a2-4be5-adfc-8ac88d69a039 None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:09:30.909351 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7467836-25a2-4be5-adfc-8ac88d69a039 None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:09:30.914151 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bee9f6a9-3e1a-4290-9b6e-97223e1033fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:09:30.914785 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee9f6a9-3e1a-4290-9b6e-97223e1033fa admin admin] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:09:30.915059 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee9f6a9-3e1a-4290-9b6e-97223e1033fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:09:30.915324 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee9f6a9-3e1a-4290-9b6e-97223e1033fa admin admin] PATH_INFO: `/v3/projects` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:09:30.915948 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bee9f6a9-3e1a-4290-9b6e-97223e1033fa admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:09:30.917840 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bee9f6a9-3e1a-4290-9b6e-97223e1033fa admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:09:30.944508 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1286/2571] 10.222.0.36 () {60 vars in 1153 bytes} [Thu Oct 20 09:09:30 2022] POST /identity/v3/projects => generated 361 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:09:30.948545 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-445ecedf-951d-48d8-a029-888151a4bc3c admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:09:30.958840 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-445ecedf-951d-48d8-a029-888151a4bc3c admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:09:31.015966 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4462e5e-6fca-467b-b558-0350b68918db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:09:31.016572 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4462e5e-6fca-467b-b558-0350b68918db admin admin] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:09:31.016851 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4462e5e-6fca-467b-b558-0350b68918db admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:09:31.017122 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4462e5e-6fca-467b-b558-0350b68918db admin admin] PATH_INFO: `/v3/users` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:09:31.017755 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4462e5e-6fca-467b-b558-0350b68918db admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:09:31.019768 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4462e5e-6fca-467b-b558-0350b68918db admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:09:31.044771 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1286/2572] 10.222.0.36 () {60 vars in 1144 bytes} [Thu Oct 20 09:09:30 2022] POST /identity/v3/users => generated 281 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 20 09:09:31.046148 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bee9f6a9-3e1a-4290-9b6e-97223e1033fa admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:09:31.056460 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bee9f6a9-3e1a-4290-9b6e-97223e1033fa admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:09:31.061002 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a203590-0a0e-4048-b800-52f5ef532fba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:09:31.061592 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a203590-0a0e-4048-b800-52f5ef532fba admin admin] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:09:31.061872 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a203590-0a0e-4048-b800-52f5ef532fba admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:09:31.062689 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a203590-0a0e-4048-b800-52f5ef532fba admin admin] PATH_INFO: `/v3/roles` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:09:31.063463 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a203590-0a0e-4048-b800-52f5ef532fba admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:09:31.065075 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a203590-0a0e-4048-b800-52f5ef532fba admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:09:31.075062 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1287/2573] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:09:31 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:09:31.079678 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4462e5e-6fca-467b-b558-0350b68918db admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:09:31.089559 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4462e5e-6fca-467b-b558-0350b68918db admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:09:31.094307 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4390f8f-e58b-4e75-9667-76bc25429429 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:09:31.094987 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4390f8f-e58b-4e75-9667-76bc25429429 admin admin] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:09:31.095219 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4390f8f-e58b-4e75-9667-76bc25429429 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:09:31.095499 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4390f8f-e58b-4e75-9667-76bc25429429 admin admin] PATH_INFO: `/v3/roles` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:09:31.096120 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4390f8f-e58b-4e75-9667-76bc25429429 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:09:31.097724 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4390f8f-e58b-4e75-9667-76bc25429429 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:09:31.108258 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1287/2574] 10.222.0.36 () {58 vars in 1122 bytes} [Thu Oct 20 09:09:31 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 20 09:09:31.112735 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a203590-0a0e-4048-b800-52f5ef532fba admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:09:31.122595 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a203590-0a0e-4048-b800-52f5ef532fba admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:09:31.128474 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f701bfd8-67aa-4599-a824-740aaebf5318 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:09:31.129111 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f701bfd8-67aa-4599-a824-740aaebf5318 admin admin] REQUEST_METHOD: `PUT` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:09:31.129390 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f701bfd8-67aa-4599-a824-740aaebf5318 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:09:31.129665 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f701bfd8-67aa-4599-a824-740aaebf5318 admin admin] PATH_INFO: `/v3/projects/a2ba46aa8b5a40b891de17437d069cbf/users/1791b78ed0654e1eb00c80ffda37766c/roles/747fbb4dfbec43baa31ec3b7761fd4b1` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:09:31.144069 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f701bfd8-67aa-4599-a824-740aaebf5318 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a2ba46aa8b5a40b891de17437d069cbf, user_id=1791b78ed0654e1eb00c80ffda37766c, role_id=747fbb4dfbec43baa31ec3b7761fd4b1)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:09:31.146763 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f701bfd8-67aa-4599-a824-740aaebf5318 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:09:31.159448 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1288/2575] 10.222.0.36 () {60 vars in 1483 bytes} [Thu Oct 20 09:09:31 2022] PUT /identity/v3/projects/a2ba46aa8b5a40b891de17437d069cbf/users/1791b78ed0654e1eb00c80ffda37766c/roles/747fbb4dfbec43baa31ec3b7761fd4b1 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:09:31.226217 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cedc04f-3d32-4ffe-9b95-1f54fc810c4c None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:09:31.226412 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cedc04f-3d32-4ffe-9b95-1f54fc810c4c None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:09:31.226729 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cedc04f-3d32-4ffe-9b95-1f54fc810c4c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:09:31.256049 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-8cedc04f-3d32-4ffe-9b95-1f54fc810c4c None None] MFA Rules not processed for user `1791b78ed0654e1eb00c80ffda37766c`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:09:31.257248 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-8cedc04f-3d32-4ffe-9b95-1f54fc810c4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:09:31.318883 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1288/2576] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:09:31 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:09:31.330088 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f701bfd8-67aa-4599-a824-740aaebf5318 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:09:31.331805 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-f701bfd8-67aa-4599-a824-740aaebf5318 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:09:31.353578 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:09:31.400170 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f701bfd8-67aa-4599-a824-740aaebf5318 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:09:31.403794 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e1b94ba-8078-4dfc-b5ca-58b892e4f622 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7cfa6bb090de44eaa5bdf099a3096e8c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:09:31.404388 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1b94ba-8078-4dfc-b5ca-58b892e4f622 service cinder] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:09:31.404662 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1b94ba-8078-4dfc-b5ca-58b892e4f622 service cinder] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:09:31.404930 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1b94ba-8078-4dfc-b5ca-58b892e4f622 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:09:31.413269 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e1b94ba-8078-4dfc-b5ca-58b892e4f622 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:09:31.414853 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e1b94ba-8078-4dfc-b5ca-58b892e4f622 service cinder] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:09:31.424047 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1289/2577] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:09:31 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:09:36.553486 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6da3e6f-5ce7-44ae-8355-90e36546e870 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:09:36.553977 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6da3e6f-5ce7-44ae-8355-90e36546e870 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:09:36.554778 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6da3e6f-5ce7-44ae-8355-90e36546e870 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:09:36.586100 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-c6da3e6f-5ce7-44ae-8355-90e36546e870 None None] MFA Rules not processed for user `f4b0e9187ce74f7888e9e7094be5c824`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:09:36.587351 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-c6da3e6f-5ce7-44ae-8355-90e36546e870 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:09:36.592232 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:09:36.667734 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1289/2578] 10.222.0.36 () {58 vars in 1009 bytes} [Thu Oct 20 09:09:36 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 20 09:09:36.720531 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e1b94ba-8078-4dfc-b5ca-58b892e4f622 service cinder] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:09:36.721968 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-9e1b94ba-8078-4dfc-b5ca-58b892e4f622 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:09:36.805223 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e1b94ba-8078-4dfc-b5ca-58b892e4f622 service cinder] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:09:36.810947 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb7f1a8-8b49-4ed7-98ad-a60b5741aefb service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4d558918cd8b4ebea4fe8765297304db', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:09:36.811555 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb7f1a8-8b49-4ed7-98ad-a60b5741aefb service swift] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:09:36.811834 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb7f1a8-8b49-4ed7-98ad-a60b5741aefb service swift] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:09:36.812143 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb7f1a8-8b49-4ed7-98ad-a60b5741aefb service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:09:36.822170 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eb7f1a8-8b49-4ed7-98ad-a60b5741aefb service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:09:36.823825 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eb7f1a8-8b49-4ed7-98ad-a60b5741aefb service swift] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:09:36.833494 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1290/2579] 10.222.0.36 () {60 vars in 1384 bytes} [Thu Oct 20 09:09:36 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 20 09:09:37.357490 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742627e3-15b3-421a-b8bb-d346ffecd7f5 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:09:37.357802 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742627e3-15b3-421a-b8bb-d346ffecd7f5 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:09:37.358258 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742627e3-15b3-421a-b8bb-d346ffecd7f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:09:37.386688 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-742627e3-15b3-421a-b8bb-d346ffecd7f5 None None] MFA Rules not processed for user `f4b0e9187ce74f7888e9e7094be5c824`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:09:37.387927 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-742627e3-15b3-421a-b8bb-d346ffecd7f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:09:37.395773 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1290/2580] 10.222.0.36 () {58 vars in 967 bytes} [Thu Oct 20 09:09:37 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:09:37.408678 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb7f1a8-8b49-4ed7-98ad-a60b5741aefb service swift] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:09:37.419848 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb7f1a8-8b49-4ed7-98ad-a60b5741aefb service swift] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:09:37.425158 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fb0dd66-b201-4704-89f4-1c66a5b10d99 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4d558918cd8b4ebea4fe8765297304db', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:09:37.425750 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb0dd66-b201-4704-89f4-1c66a5b10d99 service swift] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:09:37.426031 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb0dd66-b201-4704-89f4-1c66a5b10d99 service swift] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:09:37.426302 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb0dd66-b201-4704-89f4-1c66a5b10d99 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:09:37.434710 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fb0dd66-b201-4704-89f4-1c66a5b10d99 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:09:37.436296 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fb0dd66-b201-4704-89f4-1c66a5b10d99 service swift] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:09:37.444453 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1291/2581] 10.222.0.36 () {60 vars in 1384 bytes} [Thu Oct 20 09:09:37 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 20 09:09:37.662814 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae48d80-2faa-465d-a030-8c8fef035110 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:09:37.663303 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae48d80-2faa-465d-a030-8c8fef035110 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:09:37.663428 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae48d80-2faa-465d-a030-8c8fef035110 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:09:37.682161 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-8ae48d80-2faa-465d-a030-8c8fef035110 None None] MFA Rules not processed for user `f4b0e9187ce74f7888e9e7094be5c824`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:09:37.683385 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-8ae48d80-2faa-465d-a030-8c8fef035110 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:09:37.690954 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1291/2582] 10.222.0.36 () {58 vars in 1009 bytes} [Thu Oct 20 09:09:37 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:09:37.702735 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fb0dd66-b201-4704-89f4-1c66a5b10d99 service swift] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:09:37.713889 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fb0dd66-b201-4704-89f4-1c66a5b10d99 service swift] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:09:37.719335 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bc26c2b-180f-4c3e-9d67-5e9fc086cae5 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4d558918cd8b4ebea4fe8765297304db', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:09:37.719929 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc26c2b-180f-4c3e-9d67-5e9fc086cae5 service swift] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:09:37.720199 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc26c2b-180f-4c3e-9d67-5e9fc086cae5 service swift] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:09:37.720467 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc26c2b-180f-4c3e-9d67-5e9fc086cae5 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:09:37.728850 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bc26c2b-180f-4c3e-9d67-5e9fc086cae5 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:09:37.730425 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bc26c2b-180f-4c3e-9d67-5e9fc086cae5 service swift] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:09:37.738791 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1292/2583] 10.222.0.36 () {60 vars in 1384 bytes} [Thu Oct 20 09:09:37 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 20 09:09:38.064290 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa97ab23-0d6a-4970-9104-65e7dfa3baf0 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:09:38.064645 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa97ab23-0d6a-4970-9104-65e7dfa3baf0 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:09:38.064991 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa97ab23-0d6a-4970-9104-65e7dfa3baf0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:09:38.084927 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-aa97ab23-0d6a-4970-9104-65e7dfa3baf0 None None] MFA Rules not processed for user `f4b0e9187ce74f7888e9e7094be5c824`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:09:38.086127 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-aa97ab23-0d6a-4970-9104-65e7dfa3baf0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:09:38.093744 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1292/2584] 10.222.0.36 () {58 vars in 967 bytes} [Thu Oct 20 09:09:38 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:09:38.105934 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bc26c2b-180f-4c3e-9d67-5e9fc086cae5 service swift] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:09:38.117277 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bc26c2b-180f-4c3e-9d67-5e9fc086cae5 service swift] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:09:38.122180 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-540bf213-6740-4a7a-88ed-1e4192bdad5e service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4d558918cd8b4ebea4fe8765297304db', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:09:38.122869 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540bf213-6740-4a7a-88ed-1e4192bdad5e service swift] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:09:38.123173 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540bf213-6740-4a7a-88ed-1e4192bdad5e service swift] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:09:38.123443 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540bf213-6740-4a7a-88ed-1e4192bdad5e service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:09:38.132053 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-540bf213-6740-4a7a-88ed-1e4192bdad5e service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:09:38.133699 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-540bf213-6740-4a7a-88ed-1e4192bdad5e service swift] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:09:38.142354 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1293/2585] 10.222.0.36 () {60 vars in 1384 bytes} [Thu Oct 20 09:09:38 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 20 09:09:42.559027 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa97ab23-0d6a-4970-9104-65e7dfa3baf0 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:09:42.560435 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-aa97ab23-0d6a-4970-9104-65e7dfa3baf0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:09:42.631937 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa97ab23-0d6a-4970-9104-65e7dfa3baf0 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:09:42.637785 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c42e0b21-b286-4f31-a7c5-5245159dd4a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:09:42.638429 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c42e0b21-b286-4f31-a7c5-5245159dd4a3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:09:42.638738 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c42e0b21-b286-4f31-a7c5-5245159dd4a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:09:42.639010 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c42e0b21-b286-4f31-a7c5-5245159dd4a3 admin admin] PATH_INFO: `/v3/users/62af90b699ca42a4aa0ac8f0edfdfd31` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:09:42.640561 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c42e0b21-b286-4f31-a7c5-5245159dd4a3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=62af90b699ca42a4aa0ac8f0edfdfd31)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:09:42.643880 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c42e0b21-b286-4f31-a7c5-5245159dd4a3 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:09:42.709733 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-c42e0b21-b286-4f31-a7c5-5245159dd4a3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '62af90b699ca42a4aa0ac8f0edfdfd31'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:09:42.731503 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-c42e0b21-b286-4f31-a7c5-5245159dd4a3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '62af90b699ca42a4aa0ac8f0edfdfd31'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:09:42.747380 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-c42e0b21-b286-4f31-a7c5-5245159dd4a3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '62af90b699ca42a4aa0ac8f0edfdfd31'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:09:42.754907 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-c42e0b21-b286-4f31-a7c5-5245159dd4a3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '62af90b699ca42a4aa0ac8f0edfdfd31'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:09:42.755735 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-c42e0b21-b286-4f31-a7c5-5245159dd4a3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '62af90b699ca42a4aa0ac8f0edfdfd31'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:09:42.766378 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1293/2586] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:09:42 2022] DELETE /identity/v3/users/62af90b699ca42a4aa0ac8f0edfdfd31 => generated 0 bytes in 205 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:09:42.770528 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a863a7ec-c701-408b-9b59-b840a17ff60e None None] REQUEST_METHOD: `POST` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:09:42.770528 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a863a7ec-c701-408b-9b59-b840a17ff60e None None] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:09:42.770528 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a863a7ec-c701-408b-9b59-b840a17ff60e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:09:42.791591 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.auth.core [None req-a863a7ec-c701-408b-9b59-b840a17ff60e None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81561) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:09:42.793037 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-a863a7ec-c701-408b-9b59-b840a17ff60e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:09:42.866271 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1294/2587] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:09:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:09:42.873035 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c42e0b21-b286-4f31-a7c5-5245159dd4a3 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:09:42.874465 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-c42e0b21-b286-4f31-a7c5-5245159dd4a3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:09:42.884841 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:09:42.939373 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c42e0b21-b286-4f31-a7c5-5245159dd4a3 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:09:42.943987 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a9e9b5a-7e85-4e5e-be89-a4066cb278b7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:09:42.944720 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9e9b5a-7e85-4e5e-be89-a4066cb278b7 service neutron] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:09:42.945049 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9e9b5a-7e85-4e5e-be89-a4066cb278b7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:09:42.945469 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9e9b5a-7e85-4e5e-be89-a4066cb278b7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:09:42.958459 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a9e9b5a-7e85-4e5e-be89-a4066cb278b7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:09:42.960167 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a9e9b5a-7e85-4e5e-be89-a4066cb278b7 service neutron] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:09:42.971294 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1294/2588] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:09:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:09:43.160367 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a863a7ec-c701-408b-9b59-b840a17ff60e None None] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:09:43.162144 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-a863a7ec-c701-408b-9b59-b840a17ff60e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:09:43.177869 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a863a7ec-c701-408b-9b59-b840a17ff60e None None] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:09:43.183123 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0548f73-807c-442a-9289-3cd3266bf93e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:09:43.184543 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0548f73-807c-442a-9289-3cd3266bf93e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:09:43.184822 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0548f73-807c-442a-9289-3cd3266bf93e admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:09:43.185255 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0548f73-807c-442a-9289-3cd3266bf93e admin admin] PATH_INFO: `/v3/projects/0ce71f46acb24cd899e0d08974c825ab` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:09:43.186638 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0548f73-807c-442a-9289-3cd3266bf93e admin admin] RBAC: Authorizing `identity:delete_project(project_id=0ce71f46acb24cd899e0d08974c825ab)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:09:43.188540 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0548f73-807c-442a-9289-3cd3266bf93e admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:09:43.199266 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:09:43.244145 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-b0548f73-807c-442a-9289-3cd3266bf93e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0ce71f46acb24cd899e0d08974c825ab', 'request_id': 'req-b0548f73-807c-442a-9289-3cd3266bf93e'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:09:43.251171 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-b0548f73-807c-442a-9289-3cd3266bf93e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0ce71f46acb24cd899e0d08974c825ab', 'request_id': 'req-b0548f73-807c-442a-9289-3cd3266bf93e'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:09:43.257408 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-b0548f73-807c-442a-9289-3cd3266bf93e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0ce71f46acb24cd899e0d08974c825ab', 'request_id': 'req-b0548f73-807c-442a-9289-3cd3266bf93e'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:09:43.270231 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-b0548f73-807c-442a-9289-3cd3266bf93e admin admin] The token cache is being invalidate because project 0ce71f46acb24cd899e0d08974c825ab was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81561) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:09:43.270548 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-b0548f73-807c-442a-9289-3cd3266bf93e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:09:43.271253 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-b0548f73-807c-442a-9289-3cd3266bf93e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:09:43.272733 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1295/2589] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:09:43 2022] DELETE /identity/v3/projects/0ce71f46acb24cd899e0d08974c825ab => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:09:44.337876 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a9e9b5a-7e85-4e5e-be89-a4066cb278b7 service neutron] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:09:44.358505 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a9e9b5a-7e85-4e5e-be89-a4066cb278b7 service neutron] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:09:44.428197 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3644b76-6ba8-4e70-9525-88cd20e1f2a7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:09:44.428804 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3644b76-6ba8-4e70-9525-88cd20e1f2a7 service neutron] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:09:44.429085 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3644b76-6ba8-4e70-9525-88cd20e1f2a7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:09:44.429358 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3644b76-6ba8-4e70-9525-88cd20e1f2a7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:09:44.431737 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-f3644b76-6ba8-4e70-9525-88cd20e1f2a7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:09:44.495161 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3644b76-6ba8-4e70-9525-88cd20e1f2a7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:09:44.496592 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3644b76-6ba8-4e70-9525-88cd20e1f2a7 service neutron] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:09:44.506224 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1295/2590] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:09:44 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 169 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:09:53.266974 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=21,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:09:54.502121 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=73 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:10:06.780248 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0548f73-807c-442a-9289-3cd3266bf93e admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:10:06.781691 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-b0548f73-807c-442a-9289-3cd3266bf93e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:10:06.846304 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0548f73-807c-442a-9289-3cd3266bf93e admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:10:06.850843 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a83267fc-c200-422f-b58f-02815a984297 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:10:06.851465 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83267fc-c200-422f-b58f-02815a984297 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:10:06.851744 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83267fc-c200-422f-b58f-02815a984297 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:10:06.852016 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83267fc-c200-422f-b58f-02815a984297 admin admin] PATH_INFO: `/v3/users/1791b78ed0654e1eb00c80ffda37766c` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:10:06.853132 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a83267fc-c200-422f-b58f-02815a984297 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1791b78ed0654e1eb00c80ffda37766c)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:10:06.855133 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a83267fc-c200-422f-b58f-02815a984297 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:10:06.900162 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-a83267fc-c200-422f-b58f-02815a984297 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1791b78ed0654e1eb00c80ffda37766c'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:10:06.907149 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-a83267fc-c200-422f-b58f-02815a984297 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1791b78ed0654e1eb00c80ffda37766c'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:10:06.918751 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-a83267fc-c200-422f-b58f-02815a984297 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1791b78ed0654e1eb00c80ffda37766c'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:10:06.925512 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-a83267fc-c200-422f-b58f-02815a984297 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1791b78ed0654e1eb00c80ffda37766c'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:10:06.926224 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-a83267fc-c200-422f-b58f-02815a984297 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1791b78ed0654e1eb00c80ffda37766c'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:10:06.932169 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1296/2591] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:10:06 2022] DELETE /identity/v3/users/1791b78ed0654e1eb00c80ffda37766c => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:10:06.936564 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9565c3a3-9a45-4bd6-953a-25775c648c26 None None] REQUEST_METHOD: `POST` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:10:06.937069 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9565c3a3-9a45-4bd6-953a-25775c648c26 None None] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:10:06.937179 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9565c3a3-9a45-4bd6-953a-25775c648c26 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:10:06.957835 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.auth.core [None req-9565c3a3-9a45-4bd6-953a-25775c648c26 None None] MFA Rules not processed for user `9804008bc9774f7b96bc93c72ddebd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81562) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 20 09:10:06.959065 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-9565c3a3-9a45-4bd6-953a-25775c648c26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:10:07.015156 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1296/2592] 10.222.0.36 () {58 vars in 958 bytes} [Thu Oct 20 09:10:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 20 09:10:07.025236 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a83267fc-c200-422f-b58f-02815a984297 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:10:07.026650 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-a83267fc-c200-422f-b58f-02815a984297 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:10:07.083084 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a83267fc-c200-422f-b58f-02815a984297 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:10:07.086643 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37ed603f-f34a-4044-be17-40f28db471c2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:10:07.087233 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ed603f-f34a-4044-be17-40f28db471c2 service neutron] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:10:07.087504 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ed603f-f34a-4044-be17-40f28db471c2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:10:07.087767 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ed603f-f34a-4044-be17-40f28db471c2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:10:07.096831 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37ed603f-f34a-4044-be17-40f28db471c2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:10:07.098250 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37ed603f-f34a-4044-be17-40f28db471c2 service neutron] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:10:07.107679 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1297/2593] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:10:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:10:07.369279 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9565c3a3-9a45-4bd6-953a-25775c648c26 None None] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:10:07.370792 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-9565c3a3-9a45-4bd6-953a-25775c648c26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:10:07.383595 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9565c3a3-9a45-4bd6-953a-25775c648c26 None None] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:10:07.388231 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db2464dc-9b03-428b-836b-65fecb6009f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:10:07.389758 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db2464dc-9b03-428b-836b-65fecb6009f6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:10:07.390060 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db2464dc-9b03-428b-836b-65fecb6009f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:10:07.390718 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db2464dc-9b03-428b-836b-65fecb6009f6 admin admin] PATH_INFO: `/v3/projects/a2ba46aa8b5a40b891de17437d069cbf` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:10:07.391841 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db2464dc-9b03-428b-836b-65fecb6009f6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a2ba46aa8b5a40b891de17437d069cbf)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:10:07.394528 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db2464dc-9b03-428b-836b-65fecb6009f6 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:10:07.442875 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-db2464dc-9b03-428b-836b-65fecb6009f6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a2ba46aa8b5a40b891de17437d069cbf', 'request_id': 'req-db2464dc-9b03-428b-836b-65fecb6009f6'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:10:07.449018 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-db2464dc-9b03-428b-836b-65fecb6009f6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a2ba46aa8b5a40b891de17437d069cbf', 'request_id': 'req-db2464dc-9b03-428b-836b-65fecb6009f6'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:10:07.454575 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-db2464dc-9b03-428b-836b-65fecb6009f6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a2ba46aa8b5a40b891de17437d069cbf', 'request_id': 'req-db2464dc-9b03-428b-836b-65fecb6009f6'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:10:07.467040 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-db2464dc-9b03-428b-836b-65fecb6009f6 admin admin] The token cache is being invalidate because project a2ba46aa8b5a40b891de17437d069cbf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81562) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:10:07.467321 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-db2464dc-9b03-428b-836b-65fecb6009f6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:10:07.467964 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-db2464dc-9b03-428b-836b-65fecb6009f6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:10:07.469426 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1297/2594] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:10:07 2022] DELETE /identity/v3/projects/a2ba46aa8b5a40b891de17437d069cbf => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:10:17.103435 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=79,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:10:17.463374 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=54,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:10:34.872059 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37ed603f-f34a-4044-be17-40f28db471c2 service neutron] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:10:34.873492 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-37ed603f-f34a-4044-be17-40f28db471c2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:10:34.884733 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37ed603f-f34a-4044-be17-40f28db471c2 service neutron] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:10:34.932599 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caf7125d-9c42-4470-9b1a-362ad0716852 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:10:34.933194 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caf7125d-9c42-4470-9b1a-362ad0716852 service neutron] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:10:34.933469 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caf7125d-9c42-4470-9b1a-362ad0716852 service neutron] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:10:34.933735 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caf7125d-9c42-4470-9b1a-362ad0716852 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:10:34.935461 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-caf7125d-9c42-4470-9b1a-362ad0716852 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:10:35.014676 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caf7125d-9c42-4470-9b1a-362ad0716852 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:10:35.016029 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caf7125d-9c42-4470-9b1a-362ad0716852 service neutron] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:10:35.036693 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1298/2595] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:10:34 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 165 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:10:36.219021 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db2464dc-9b03-428b-836b-65fecb6009f6 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:10:36.220936 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-db2464dc-9b03-428b-836b-65fecb6009f6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:10:36.233010 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db2464dc-9b03-428b-836b-65fecb6009f6 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:10:36.237150 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aaf742a-366b-465a-98d5-d7a1c54dc163 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:10:36.237733 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aaf742a-366b-465a-98d5-d7a1c54dc163 service neutron] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:10:36.238007 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aaf742a-366b-465a-98d5-d7a1c54dc163 service neutron] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:10:36.238274 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aaf742a-366b-465a-98d5-d7a1c54dc163 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:10:36.239952 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-2aaf742a-366b-465a-98d5-d7a1c54dc163 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:10:36.252793 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aaf742a-366b-465a-98d5-d7a1c54dc163 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:10:36.254200 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aaf742a-366b-465a-98d5-d7a1c54dc163 service neutron] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:10:36.264918 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1298/2596] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:10:36 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:10:45.034120 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=63 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:10:46.260019 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=6 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:11:03.365040 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caf7125d-9c42-4470-9b1a-362ad0716852 service neutron] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:11:03.366527 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-caf7125d-9c42-4470-9b1a-362ad0716852 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:11:03.377859 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caf7125d-9c42-4470-9b1a-362ad0716852 service neutron] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:11:03.381553 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef7e7012-d2bc-4f68-bc6e-b5c543e32a0b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:11:03.382172 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7e7012-d2bc-4f68-bc6e-b5c543e32a0b service neutron] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:11:03.382447 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7e7012-d2bc-4f68-bc6e-b5c543e32a0b service neutron] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:11:03.382742 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7e7012-d2bc-4f68-bc6e-b5c543e32a0b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:11:03.384469 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-ef7e7012-d2bc-4f68-bc6e-b5c543e32a0b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:11:03.394763 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef7e7012-d2bc-4f68-bc6e-b5c543e32a0b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:11:03.396186 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef7e7012-d2bc-4f68-bc6e-b5c543e32a0b service neutron] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:11:03.404364 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1299/2597] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:11:03 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:11:13.400647 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=6 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:11:26.232057 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aaf742a-366b-465a-98d5-d7a1c54dc163 service neutron] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:11:26.233624 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-2aaf742a-366b-465a-98d5-d7a1c54dc163 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:11:26.248948 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aaf742a-366b-465a-98d5-d7a1c54dc163 service neutron] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:11:26.254530 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd3e0227-9fae-46a0-b647-066a736ea5f0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a94e868115ec49918eda10724263930c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:11:26.255215 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd3e0227-9fae-46a0-b647-066a736ea5f0 service nova] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:11:26.256079 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd3e0227-9fae-46a0-b647-066a736ea5f0 service nova] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:11:26.256079 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd3e0227-9fae-46a0-b647-066a736ea5f0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:11:26.257663 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-bd3e0227-9fae-46a0-b647-066a736ea5f0 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:11:26.270343 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd3e0227-9fae-46a0-b647-066a736ea5f0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:11:26.272009 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd3e0227-9fae-46a0-b647-066a736ea5f0 service nova] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:11:26.283442 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1299/2598] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:11:26 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:11:28.881965 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef7e7012-d2bc-4f68-bc6e-b5c543e32a0b service neutron] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:11:28.883846 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-ef7e7012-d2bc-4f68-bc6e-b5c543e32a0b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:11:28.904290 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef7e7012-d2bc-4f68-bc6e-b5c543e32a0b service neutron] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:11:28.910612 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be148897-ef7c-46ae-b7ea-a4bfa3f43549 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a94e868115ec49918eda10724263930c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'ResellerAdmin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:11:28.911207 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be148897-ef7c-46ae-b7ea-a4bfa3f43549 service nova] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:11:28.911482 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be148897-ef7c-46ae-b7ea-a4bfa3f43549 service nova] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:11:28.911746 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be148897-ef7c-46ae-b7ea-a4bfa3f43549 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:11:28.914316 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-be148897-ef7c-46ae-b7ea-a4bfa3f43549 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:11:28.928807 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be148897-ef7c-46ae-b7ea-a4bfa3f43549 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:11:28.930758 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be148897-ef7c-46ae-b7ea-a4bfa3f43549 service nova] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:11:28.942084 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1300/2599] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:11:28 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 60 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:11:36.278879 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=6 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:11:38.937291 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=6 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:11:39.618058 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd3e0227-9fae-46a0-b647-066a736ea5f0 service nova] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:11:39.628097 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd3e0227-9fae-46a0-b647-066a736ea5f0 service nova] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:11:39.631657 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b234bb7c-4c53-4bb5-944b-507c89f7fa0c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:11:39.632176 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b234bb7c-4c53-4bb5-944b-507c89f7fa0c service neutron] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:11:39.632433 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b234bb7c-4c53-4bb5-944b-507c89f7fa0c service neutron] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:11:39.632692 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b234bb7c-4c53-4bb5-944b-507c89f7fa0c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:11:39.634348 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-b234bb7c-4c53-4bb5-944b-507c89f7fa0c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:11:39.695310 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b234bb7c-4c53-4bb5-944b-507c89f7fa0c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:11:39.696731 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b234bb7c-4c53-4bb5-944b-507c89f7fa0c service neutron] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:11:39.706477 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1300/2600] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:11:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:11:45.768478 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be148897-ef7c-46ae-b7ea-a4bfa3f43549 service nova] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:11:45.770001 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-be148897-ef7c-46ae-b7ea-a4bfa3f43549 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:11:45.785579 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be148897-ef7c-46ae-b7ea-a4bfa3f43549 service nova] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:11:45.790091 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29d7ca9d-807a-4a92-8658-d31b651422e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:11:45.790743 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29d7ca9d-807a-4a92-8658-d31b651422e0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:11:45.791021 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29d7ca9d-807a-4a92-8658-d31b651422e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:11:45.791293 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29d7ca9d-807a-4a92-8658-d31b651422e0 admin admin] PATH_INFO: `/v3/users/65ca2eae6db24bb094afaacdfb5c1980` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:11:45.792396 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29d7ca9d-807a-4a92-8658-d31b651422e0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=65ca2eae6db24bb094afaacdfb5c1980)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:11:45.794322 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29d7ca9d-807a-4a92-8658-d31b651422e0 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:11:45.845107 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-29d7ca9d-807a-4a92-8658-d31b651422e0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '65ca2eae6db24bb094afaacdfb5c1980'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:11:45.853477 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-29d7ca9d-807a-4a92-8658-d31b651422e0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '65ca2eae6db24bb094afaacdfb5c1980'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:11:45.866205 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-29d7ca9d-807a-4a92-8658-d31b651422e0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '65ca2eae6db24bb094afaacdfb5c1980'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:11:45.876961 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-29d7ca9d-807a-4a92-8658-d31b651422e0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '65ca2eae6db24bb094afaacdfb5c1980'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:11:45.877613 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-29d7ca9d-807a-4a92-8658-d31b651422e0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '65ca2eae6db24bb094afaacdfb5c1980'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:11:45.884184 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1301/2601] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:11:45 2022] DELETE /identity/v3/users/65ca2eae6db24bb094afaacdfb5c1980 => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:11:46.070171 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b234bb7c-4c53-4bb5-944b-507c89f7fa0c service neutron] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:11:46.082991 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b234bb7c-4c53-4bb5-944b-507c89f7fa0c service neutron] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:11:46.140949 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3cf711-b4fa-4d6c-a102-141788fdf31a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:11:46.141569 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3cf711-b4fa-4d6c-a102-141788fdf31a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:11:46.141842 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3cf711-b4fa-4d6c-a102-141788fdf31a admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:11:46.142110 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3cf711-b4fa-4d6c-a102-141788fdf31a admin admin] PATH_INFO: `/v3/projects/98827dbef155480b9f26792b70014b4b` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:11:46.143288 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff3cf711-b4fa-4d6c-a102-141788fdf31a admin admin] RBAC: Authorizing `identity:delete_project(project_id=98827dbef155480b9f26792b70014b4b)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:11:46.145161 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff3cf711-b4fa-4d6c-a102-141788fdf31a admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:11:46.191385 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-ff3cf711-b4fa-4d6c-a102-141788fdf31a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '98827dbef155480b9f26792b70014b4b', 'request_id': 'req-ff3cf711-b4fa-4d6c-a102-141788fdf31a'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:11:46.197441 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-ff3cf711-b4fa-4d6c-a102-141788fdf31a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '98827dbef155480b9f26792b70014b4b', 'request_id': 'req-ff3cf711-b4fa-4d6c-a102-141788fdf31a'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:11:46.203468 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-ff3cf711-b4fa-4d6c-a102-141788fdf31a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '98827dbef155480b9f26792b70014b4b', 'request_id': 'req-ff3cf711-b4fa-4d6c-a102-141788fdf31a'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:11:46.216740 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-ff3cf711-b4fa-4d6c-a102-141788fdf31a admin admin] The token cache is being invalidate because project 98827dbef155480b9f26792b70014b4b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81562) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:11:46.217023 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-ff3cf711-b4fa-4d6c-a102-141788fdf31a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:11:46.218259 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-ff3cf711-b4fa-4d6c-a102-141788fdf31a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:11:46.219851 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1301/2602] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:11:46 2022] DELETE /identity/v3/projects/98827dbef155480b9f26792b70014b4b => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:11:46.223807 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29d7ca9d-807a-4a92-8658-d31b651422e0 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:11:46.225318 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-29d7ca9d-807a-4a92-8658-d31b651422e0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:11:46.238999 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29d7ca9d-807a-4a92-8658-d31b651422e0 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:11:46.293937 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93da0d8a-e0a3-4d33-8e3b-8d5f11f9c2ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:11:46.294595 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93da0d8a-e0a3-4d33-8e3b-8d5f11f9c2ad admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:11:46.294877 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93da0d8a-e0a3-4d33-8e3b-8d5f11f9c2ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:11:46.295147 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93da0d8a-e0a3-4d33-8e3b-8d5f11f9c2ad admin admin] PATH_INFO: `/v3/users/8b06169f2edd4818ab2382e76c3632ac` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:11:46.296805 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93da0d8a-e0a3-4d33-8e3b-8d5f11f9c2ad admin admin] RBAC: Authorizing `identity:delete_user(user_id=8b06169f2edd4818ab2382e76c3632ac)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:11:46.298771 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93da0d8a-e0a3-4d33-8e3b-8d5f11f9c2ad admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:11:46.343881 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-93da0d8a-e0a3-4d33-8e3b-8d5f11f9c2ad admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8b06169f2edd4818ab2382e76c3632ac'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:11:46.350987 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-93da0d8a-e0a3-4d33-8e3b-8d5f11f9c2ad admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8b06169f2edd4818ab2382e76c3632ac'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:11:46.363666 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-93da0d8a-e0a3-4d33-8e3b-8d5f11f9c2ad admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8b06169f2edd4818ab2382e76c3632ac'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:11:46.369134 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=78,keystone:DELETE=20,keystone:INSERT=2 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:11:46.377438 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-93da0d8a-e0a3-4d33-8e3b-8d5f11f9c2ad admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8b06169f2edd4818ab2382e76c3632ac'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:11:46.378495 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-93da0d8a-e0a3-4d33-8e3b-8d5f11f9c2ad admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8b06169f2edd4818ab2382e76c3632ac'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:11:46.384881 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1302/2603] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:11:46 2022] DELETE /identity/v3/users/8b06169f2edd4818ab2382e76c3632ac => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:11:46.558090 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3cf711-b4fa-4d6c-a102-141788fdf31a admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:11:46.559791 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-ff3cf711-b4fa-4d6c-a102-141788fdf31a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:11:46.574691 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3cf711-b4fa-4d6c-a102-141788fdf31a admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:11:46.596336 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:11:46.634883 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd15362e-e747-40bd-9b70-33eea9409d3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:11:46.635491 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd15362e-e747-40bd-9b70-33eea9409d3e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:11:46.635769 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd15362e-e747-40bd-9b70-33eea9409d3e admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:11:46.636041 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd15362e-e747-40bd-9b70-33eea9409d3e admin admin] PATH_INFO: `/v3/projects/630293d0e194481c86354df95acdd799` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:11:46.637156 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd15362e-e747-40bd-9b70-33eea9409d3e admin admin] RBAC: Authorizing `identity:delete_project(project_id=630293d0e194481c86354df95acdd799)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:11:46.639109 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd15362e-e747-40bd-9b70-33eea9409d3e admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:11:46.684447 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-bd15362e-e747-40bd-9b70-33eea9409d3e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '630293d0e194481c86354df95acdd799', 'request_id': 'req-bd15362e-e747-40bd-9b70-33eea9409d3e'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:11:46.690733 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-bd15362e-e747-40bd-9b70-33eea9409d3e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '630293d0e194481c86354df95acdd799', 'request_id': 'req-bd15362e-e747-40bd-9b70-33eea9409d3e'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:11:46.696274 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-bd15362e-e747-40bd-9b70-33eea9409d3e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '630293d0e194481c86354df95acdd799', 'request_id': 'req-bd15362e-e747-40bd-9b70-33eea9409d3e'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:11:46.708791 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-bd15362e-e747-40bd-9b70-33eea9409d3e admin admin] The token cache is being invalidate because project 630293d0e194481c86354df95acdd799 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81562) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:11:46.709075 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-bd15362e-e747-40bd-9b70-33eea9409d3e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:11:46.709689 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-bd15362e-e747-40bd-9b70-33eea9409d3e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:11:46.711249 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1302/2604] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:11:46 2022] DELETE /identity/v3/projects/630293d0e194481c86354df95acdd799 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:11:56.381531 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=4 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:11:56.705590 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=40,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:12:27.838854 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93da0d8a-e0a3-4d33-8e3b-8d5f11f9c2ad admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:12:27.840555 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-93da0d8a-e0a3-4d33-8e3b-8d5f11f9c2ad admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:12:27.899142 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93da0d8a-e0a3-4d33-8e3b-8d5f11f9c2ad admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:12:27.946153 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ecd1ca6-4c74-4bc6-b5f0-634802fa1df4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '794ca389002d41748d7925216ed964a5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:12:27.946842 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ecd1ca6-4c74-4bc6-b5f0-634802fa1df4 service neutron] REQUEST_METHOD: `GET` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:12:27.947117 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ecd1ca6-4c74-4bc6-b5f0-634802fa1df4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:12:27.947389 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ecd1ca6-4c74-4bc6-b5f0-634802fa1df4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:12:27.949089 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-0ecd1ca6-4c74-4bc6-b5f0-634802fa1df4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:12:28.010515 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ecd1ca6-4c74-4bc6-b5f0-634802fa1df4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:12:28.011950 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ecd1ca6-4c74-4bc6-b5f0-634802fa1df4 service neutron] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:12:28.021687 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1303/2605] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:12:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 183 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:12:34.292754 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd15362e-e747-40bd-9b70-33eea9409d3e admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:12:34.294273 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-bd15362e-e747-40bd-9b70-33eea9409d3e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:12:34.310423 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd15362e-e747-40bd-9b70-33eea9409d3e admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:12:34.317568 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61081eff-b0d7-4aaf-a724-0551b5bd321c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:12:34.318285 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61081eff-b0d7-4aaf-a724-0551b5bd321c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:12:34.318659 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61081eff-b0d7-4aaf-a724-0551b5bd321c admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:12:34.318993 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61081eff-b0d7-4aaf-a724-0551b5bd321c admin admin] PATH_INFO: `/v3/users/21311bc5eef742f5a9eb81462a701ceb` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:12:34.320523 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61081eff-b0d7-4aaf-a724-0551b5bd321c admin admin] RBAC: Authorizing `identity:delete_user(user_id=21311bc5eef742f5a9eb81462a701ceb)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:12:34.322845 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61081eff-b0d7-4aaf-a724-0551b5bd321c admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:12:34.371366 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-61081eff-b0d7-4aaf-a724-0551b5bd321c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '21311bc5eef742f5a9eb81462a701ceb'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:12:34.379934 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-61081eff-b0d7-4aaf-a724-0551b5bd321c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '21311bc5eef742f5a9eb81462a701ceb'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:12:34.392912 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-61081eff-b0d7-4aaf-a724-0551b5bd321c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '21311bc5eef742f5a9eb81462a701ceb'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:12:34.399671 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-61081eff-b0d7-4aaf-a724-0551b5bd321c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '21311bc5eef742f5a9eb81462a701ceb'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:12:34.400353 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-61081eff-b0d7-4aaf-a724-0551b5bd321c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '21311bc5eef742f5a9eb81462a701ceb'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:12:34.406412 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1303/2606] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:12:34 2022] DELETE /identity/v3/users/21311bc5eef742f5a9eb81462a701ceb => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:12:34.574423 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ecd1ca6-4c74-4bc6-b5f0-634802fa1df4 service neutron] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:12:34.584762 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ecd1ca6-4c74-4bc6-b5f0-634802fa1df4 service neutron] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:12:34.641284 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-011ecfe5-61ab-4135-a818-c0c869248190 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:12:34.641904 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-011ecfe5-61ab-4135-a818-c0c869248190 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:12:34.642181 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-011ecfe5-61ab-4135-a818-c0c869248190 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:12:34.642448 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-011ecfe5-61ab-4135-a818-c0c869248190 admin admin] PATH_INFO: `/v3/projects/45d1505e085847329f7480c9bc08f1d8` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:12:34.643563 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-011ecfe5-61ab-4135-a818-c0c869248190 admin admin] RBAC: Authorizing `identity:delete_project(project_id=45d1505e085847329f7480c9bc08f1d8)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:12:34.645499 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-011ecfe5-61ab-4135-a818-c0c869248190 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:12:34.653291 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=100 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:12:34.691249 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-011ecfe5-61ab-4135-a818-c0c869248190 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '45d1505e085847329f7480c9bc08f1d8', 'request_id': 'req-011ecfe5-61ab-4135-a818-c0c869248190'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:12:34.697298 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-011ecfe5-61ab-4135-a818-c0c869248190 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '45d1505e085847329f7480c9bc08f1d8', 'request_id': 'req-011ecfe5-61ab-4135-a818-c0c869248190'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:12:34.702975 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-011ecfe5-61ab-4135-a818-c0c869248190 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '45d1505e085847329f7480c9bc08f1d8', 'request_id': 'req-011ecfe5-61ab-4135-a818-c0c869248190'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:12:34.716537 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-011ecfe5-61ab-4135-a818-c0c869248190 admin admin] The token cache is being invalidate because project 45d1505e085847329f7480c9bc08f1d8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81561) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:12:34.716826 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-011ecfe5-61ab-4135-a818-c0c869248190 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:12:34.717447 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-011ecfe5-61ab-4135-a818-c0c869248190 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:12:34.718982 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1304/2607] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:12:34 2022] DELETE /identity/v3/projects/45d1505e085847329f7480c9bc08f1d8 => generated 0 bytes in 145 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:12:34.723555 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61081eff-b0d7-4aaf-a724-0551b5bd321c admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:12:34.724982 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-61081eff-b0d7-4aaf-a724-0551b5bd321c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:12:34.739096 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61081eff-b0d7-4aaf-a724-0551b5bd321c admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:12:34.795635 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-598b83bb-9589-4a8b-a03e-221470f41ad7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:12:34.796265 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598b83bb-9589-4a8b-a03e-221470f41ad7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:12:34.796543 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598b83bb-9589-4a8b-a03e-221470f41ad7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:12:34.796818 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598b83bb-9589-4a8b-a03e-221470f41ad7 admin admin] PATH_INFO: `/v3/users/b75a8da9cea842ec8a1eff8314e5d46f` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:12:34.798396 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-598b83bb-9589-4a8b-a03e-221470f41ad7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b75a8da9cea842ec8a1eff8314e5d46f)` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:12:34.800346 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-598b83bb-9589-4a8b-a03e-221470f41ad7 admin admin] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:12:34.846737 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-598b83bb-9589-4a8b-a03e-221470f41ad7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b75a8da9cea842ec8a1eff8314e5d46f'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:12:34.853817 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-598b83bb-9589-4a8b-a03e-221470f41ad7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b75a8da9cea842ec8a1eff8314e5d46f'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:12:34.866712 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-598b83bb-9589-4a8b-a03e-221470f41ad7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b75a8da9cea842ec8a1eff8314e5d46f'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:12:34.868774 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=78,keystone:DELETE=20,keystone:INSERT=2 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:12:34.874932 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-598b83bb-9589-4a8b-a03e-221470f41ad7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b75a8da9cea842ec8a1eff8314e5d46f'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:12:34.875985 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.notifications [None req-598b83bb-9589-4a8b-a03e-221470f41ad7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b75a8da9cea842ec8a1eff8314e5d46f'} {{(pid=81562) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:12:34.882123 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1304/2608] 10.222.0.36 () {58 vars in 1224 bytes} [Thu Oct 20 09:12:34 2022] DELETE /identity/v3/users/b75a8da9cea842ec8a1eff8314e5d46f => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 20 09:12:35.051279 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-011ecfe5-61ab-4135-a818-c0c869248190 admin admin] Authenticating user token {{(pid=81561) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:12:35.052788 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.fernet_utils [None req-011ecfe5-61ab-4135-a818-c0c869248190 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81561) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:12:35.066685 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-011ecfe5-61ab-4135-a818-c0c869248190 admin admin] Validating token access rules against request {{(pid=81561) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:12:35.124501 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8941ee1-da49-4303-bbbb-d37964370c65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9804008bc9774f7b96bc93c72ddebd2f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '602743e1ebcd495bbe3a73b9f8abba4e', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81561) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:12:35.125121 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8941ee1-da49-4303-bbbb-d37964370c65 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:12:35.125398 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8941ee1-da49-4303-bbbb-d37964370c65 admin admin] SCRIPT_NAME: `/identity` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:12:35.126232 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8941ee1-da49-4303-bbbb-d37964370c65 admin admin] PATH_INFO: `/v3/projects/e216ca9a35ba499998cd5a7a1a8b37c7` {{(pid=81561) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:12:35.127355 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8941ee1-da49-4303-bbbb-d37964370c65 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e216ca9a35ba499998cd5a7a1a8b37c7)` {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:12:35.129260 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8941ee1-da49-4303-bbbb-d37964370c65 admin admin] RBAC: Authorization granted {{(pid=81561) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:12:35.174913 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-f8941ee1-da49-4303-bbbb-d37964370c65 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e216ca9a35ba499998cd5a7a1a8b37c7', 'request_id': 'req-f8941ee1-da49-4303-bbbb-d37964370c65'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:12:35.181036 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-f8941ee1-da49-4303-bbbb-d37964370c65 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e216ca9a35ba499998cd5a7a1a8b37c7', 'request_id': 'req-f8941ee1-da49-4303-bbbb-d37964370c65'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:12:35.186390 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-f8941ee1-da49-4303-bbbb-d37964370c65 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e216ca9a35ba499998cd5a7a1a8b37c7', 'request_id': 'req-f8941ee1-da49-4303-bbbb-d37964370c65'} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:12:35.198581 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-f8941ee1-da49-4303-bbbb-d37964370c65 admin admin] The token cache is being invalidate because project e216ca9a35ba499998cd5a7a1a8b37c7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81561) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 20 09:12:35.198863 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-f8941ee1-da49-4303-bbbb-d37964370c65 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:12:35.199482 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG keystone.notifications [None req-f8941ee1-da49-4303-bbbb-d37964370c65 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81561) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 20 09:12:35.201120 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: [pid: 81561|app: 0|req: 1305/2609] 10.222.0.36 () {58 vars in 1233 bytes} [Thu Oct 20 09:12:35 2022] DELETE /identity/v3/projects/e216ca9a35ba499998cd5a7a1a8b37c7 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 20 09:12:44.878761 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=4 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:12:45.195590 dv-cinder-master-3-iscsi devstack@keystone.service[81561]: DEBUG dbcounter [-] [81561] Writing DB stats keystone:SELECT=72,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81561) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 20 09:14:47.796218 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-598b83bb-9589-4a8b-a03e-221470f41ad7 admin admin] Authenticating user token {{(pid=81562) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 20 09:14:47.797625 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-598b83bb-9589-4a8b-a03e-221470f41ad7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:14:47.866055 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-598b83bb-9589-4a8b-a03e-221470f41ad7 admin admin] Validating token access rules against request {{(pid=81562) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 20 09:14:47.923854 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-029b731c-a2f7-43fe-9f9f-a2816c303c8e service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ad03ab76b9c44b1cb369475afa25eddd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd9857b610fb542969cfb5efbbdb8ce61', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81562) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 20 09:14:47.924444 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029b731c-a2f7-43fe-9f9f-a2816c303c8e service placement] REQUEST_METHOD: `GET` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 20 09:14:47.924714 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029b731c-a2f7-43fe-9f9f-a2816c303c8e service placement] SCRIPT_NAME: `/identity` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 20 09:14:47.924974 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029b731c-a2f7-43fe-9f9f-a2816c303c8e service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81562) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 20 09:14:47.926669 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.fernet_utils [None req-029b731c-a2f7-43fe-9f9f-a2816c303c8e service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81562) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 20 09:14:47.937367 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-029b731c-a2f7-43fe-9f9f-a2816c303c8e service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 20 09:14:47.938938 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-029b731c-a2f7-43fe-9f9f-a2816c303c8e service placement] RBAC: Authorization granted {{(pid=81562) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 20 09:14:47.949618 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: [pid: 81562|app: 0|req: 1305/2610] 10.222.0.36 () {60 vars in 1365 bytes} [Thu Oct 20 09:14:47 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 154 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 20 09:14:57.945410 dv-cinder-master-3-iscsi devstack@keystone.service[81562]: DEBUG dbcounter [-] [81562] Writing DB stats keystone:SELECT=58 {{(pid=81562) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}